Bank Churn Prediction

Problem Statement¶

Context¶

Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.

Objective¶

You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.

Data Dictionary¶

  • CustomerId: Unique ID which is assigned to each customer

  • Surname: Last name of the customer

  • CreditScore: It defines the credit history of the customer.

  • Geography: A customer’s location

  • Gender: It defines the Gender of the customer

  • Age: Age of the customer

  • Tenure: Number of years for which the customer has been with the bank

  • NumOfProducts: refers to the number of products that a customer has purchased through the bank.

  • Balance: Account balance

  • HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.

  • EstimatedSalary: Estimated salary

  • isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )

  • Exited : whether or not the customer left the bank within six month. It can take two values ** 0=No ( Customer did not leave the bank ) ** 1=Yes ( Customer left the bank )

Importing necessary libraries¶

In [3]:
!pip install tensorflow==2.18.0 scikit-learn==1.6.0 seaborn==0.13.1 matplotlib==3.8.1 numpy==1.26.2 pandas==2.2.2 imbalanced-learn==0.13.0 -q --user
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 1.8 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.5/13.5 MB 110.4 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 294.8/294.8 kB 22.6 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 94.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 93.2 MB/s eta 0:00:00
  WARNING: The script f2py is installed in '/root/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
In [1]:
import pandas as pd  # Library for data manipulation and analysis.
import numpy as np   # Fundamental package for scientific computing.
import matplotlib.pyplot as plt  # Plotting library for creating visualizations.
import seaborn as sns #For advanced visualizations.

from sklearn.model_selection import StratifiedKFold
from sklearn.utils.class_weight import compute_class_weight
from sklearn.model_selection import train_test_split  # Function for splitting datasets for training and testing.
from sklearn.preprocessing import StandardScaler
import category_encoders as ce  # Assuming you've installed category_encoders

import time  # Module for time-related operations.

from imblearn.over_sampling import SMOTE  # Import SMOTE from the correct module
from imblearn.under_sampling import RandomUnderSampler  # Import RandomUnderSampler from the correct module
import tensorflow as tf #An end-to-end open source machine learning platform
tf.config.run_functions_eagerly(True)
from tensorflow import keras  # High-level neural networks API for deep learning.
from keras import backend as K
from keras import backend   # Abstraction layer for neural network backend engines.
from keras.models import Sequential  # Model for building NN sequentially.
from keras.layers import Dense,Dropout,BatchNormalization   # for creating fully connected neural network layers.
from keras import regularizers  # Import regularizers
from tensorflow.keras.layers import LeakyReLU
from tensorflow.keras.layers import Input # Add this line to import Input
from tensorflow.keras.models import Sequential, Model # Update this line to include Model
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization # Update this line to include BatchNormalization
from keras.callbacks import EarlyStopping
In [5]:
!pip install category-encoders==2.6.2 --user
Collecting category-encoders==2.6.2
  Downloading category_encoders-2.6.2-py2.py3-none-any.whl.metadata (8.0 kB)
Requirement already satisfied: numpy>=1.14.0 in /root/.local/lib/python3.11/site-packages (from category-encoders==2.6.2) (1.26.2)
Requirement already satisfied: scikit-learn>=0.20.0 in /root/.local/lib/python3.11/site-packages (from category-encoders==2.6.2) (1.6.0)
Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from category-encoders==2.6.2) (1.13.1)
Requirement already satisfied: statsmodels>=0.9.0 in /usr/local/lib/python3.11/dist-packages (from category-encoders==2.6.2) (0.14.4)
Requirement already satisfied: pandas>=1.0.5 in /usr/local/lib/python3.11/dist-packages (from category-encoders==2.6.2) (2.2.2)
Requirement already satisfied: patsy>=0.5.1 in /usr/local/lib/python3.11/dist-packages (from category-encoders==2.6.2) (1.0.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.0.5->category-encoders==2.6.2) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.0.5->category-encoders==2.6.2) (2025.1)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.0.5->category-encoders==2.6.2) (2025.1)
Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.11/dist-packages (from scikit-learn>=0.20.0->category-encoders==2.6.2) (1.4.2)
Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.11/dist-packages (from scikit-learn>=0.20.0->category-encoders==2.6.2) (3.5.0)
Requirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.11/dist-packages (from statsmodels>=0.9.0->category-encoders==2.6.2) (24.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.8.2->pandas>=1.0.5->category-encoders==2.6.2) (1.17.0)
Downloading category_encoders-2.6.2-py2.py3-none-any.whl (81 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.8/81.8 kB 2.3 MB/s eta 0:00:00
Installing collected packages: category-encoders
Successfully installed category-encoders-2.6.2
In [6]:
!pip install category-encoders -q --user

Loading the dataset¶

In [2]:
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [3]:
data = pd.read_csv("/content/drive/MyDrive/Neural Network Bank Project/bank-1.csv")
In [4]:
df=data.copy()

Data Overview¶

In [ ]:
df.head(25)
Out[ ]:
RowNumber CustomerId Surname CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited
0 1 15634602 Hargrave 619 France Female 42 2 0.00 1 1 1 101348.88 1
1 2 15647311 Hill 608 Spain Female 41 1 83807.86 1 0 1 112542.58 0
2 3 15619304 Onio 502 France Female 42 8 159660.80 3 1 0 113931.57 1
3 4 15701354 Boni 699 France Female 39 1 0.00 2 0 0 93826.63 0
4 5 15737888 Mitchell 850 Spain Female 43 2 125510.82 1 1 1 79084.10 0
5 6 15574012 Chu 645 Spain Male 44 8 113755.78 2 1 0 149756.71 1
6 7 15592531 Bartlett 822 France Male 50 7 0.00 2 1 1 10062.80 0
7 8 15656148 Obinna 376 Germany Female 29 4 115046.74 4 1 0 119346.88 1
8 9 15792365 He 501 France Male 44 4 142051.07 2 0 1 74940.50 0
9 10 15592389 H? 684 France Male 27 2 134603.88 1 1 1 71725.73 0
10 11 15767821 Bearce 528 France Male 31 6 102016.72 2 0 0 80181.12 0
11 12 15737173 Andrews 497 Spain Male 24 3 0.00 2 1 0 76390.01 0
12 13 15632264 Kay 476 France Female 34 10 0.00 2 1 0 26260.98 0
13 14 15691483 Chin 549 France Female 25 5 0.00 2 0 0 190857.79 0
14 15 15600882 Scott 635 Spain Female 35 7 0.00 2 1 1 65951.65 0
15 16 15643966 Goforth 616 Germany Male 45 3 143129.41 2 0 1 64327.26 0
16 17 15737452 Romeo 653 Germany Male 58 1 132602.88 1 1 0 5097.67 1
17 18 15788218 Henderson 549 Spain Female 24 9 0.00 2 1 1 14406.41 0
18 19 15661507 Muldrow 587 Spain Male 45 6 0.00 1 0 0 158684.81 0
19 20 15568982 Hao 726 France Female 24 6 0.00 2 1 1 54724.03 0
20 21 15577657 McDonald 732 France Male 41 8 0.00 2 1 1 170886.17 0
21 22 15597945 Dellucci 636 Spain Female 32 8 0.00 2 1 0 138555.46 0
22 23 15699309 Gerasimov 510 Spain Female 38 4 0.00 1 1 0 118913.53 1
23 24 15725737 Mosman 669 France Male 46 3 0.00 2 0 1 8487.75 0
24 25 15625047 Yen 846 France Female 38 5 0.00 1 1 1 187616.16 0
In [ ]:
df.tail()
Out[ ]:
RowNumber CustomerId Surname CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited
9995 9996 15606229 Obijiaku 771 France Male 39 5 0.00 2 1 0 96270.64 0
9996 9997 15569892 Johnstone 516 France Male 35 10 57369.61 1 1 1 101699.77 0
9997 9998 15584532 Liu 709 France Female 36 7 0.00 1 0 1 42085.58 1
9998 9999 15682355 Sabbatini 772 Germany Male 42 3 75075.31 2 1 0 92888.52 1
9999 10000 15628319 Walker 792 France Female 28 4 130142.79 1 1 0 38190.78 0
In [ ]:
df.shape
Out[ ]:
(10000, 14)
In [ ]:
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 14 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   RowNumber        10000 non-null  int64  
 1   CustomerId       10000 non-null  int64  
 2   Surname          10000 non-null  object 
 3   CreditScore      10000 non-null  int64  
 4   Geography        10000 non-null  object 
 5   Gender           10000 non-null  object 
 6   Age              10000 non-null  int64  
 7   Tenure           10000 non-null  int64  
 8   Balance          10000 non-null  float64
 9   NumOfProducts    10000 non-null  int64  
 10  HasCrCard        10000 non-null  int64  
 11  IsActiveMember   10000 non-null  int64  
 12  EstimatedSalary  10000 non-null  float64
 13  Exited           10000 non-null  int64  
dtypes: float64(2), int64(9), object(3)
memory usage: 1.1+ MB
In [ ]:
df.describe()
Out[ ]:
RowNumber CustomerId CreditScore Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited
count 10000.00000 1.000000e+04 10000.000000 10000.000000 10000.000000 10000.000000 10000.000000 10000.00000 10000.000000 10000.000000 10000.000000
mean 5000.50000 1.569094e+07 650.528800 38.921800 5.012800 76485.889288 1.530200 0.70550 0.515100 100090.239881 0.203700
std 2886.89568 7.193619e+04 96.653299 10.487806 2.892174 62397.405202 0.581654 0.45584 0.499797 57510.492818 0.402769
min 1.00000 1.556570e+07 350.000000 18.000000 0.000000 0.000000 1.000000 0.00000 0.000000 11.580000 0.000000
25% 2500.75000 1.562853e+07 584.000000 32.000000 3.000000 0.000000 1.000000 0.00000 0.000000 51002.110000 0.000000
50% 5000.50000 1.569074e+07 652.000000 37.000000 5.000000 97198.540000 1.000000 1.00000 1.000000 100193.915000 0.000000
75% 7500.25000 1.575323e+07 718.000000 44.000000 7.000000 127644.240000 2.000000 1.00000 1.000000 149388.247500 0.000000
max 10000.00000 1.581569e+07 850.000000 92.000000 10.000000 250898.090000 4.000000 1.00000 1.000000 199992.480000 1.000000
In [ ]:
df.describe(include=["object"]).T
Out[ ]:
count unique top freq
Surname 10000 2932 Smith 32
Geography 10000 3 France 5014
Gender 10000 2 Male 5457
In [ ]:
df.isnull().sum()
Out[ ]:
0
RowNumber 0
CustomerId 0
Surname 0
CreditScore 0
Geography 0
Gender 0
Age 0
Tenure 0
Balance 0
NumOfProducts 0
HasCrCard 0
IsActiveMember 0
EstimatedSalary 0
Exited 0

In [5]:
df.drop("RowNumber", axis=1, inplace=True)
In [6]:
df.drop("CustomerId", axis=1, inplace=True)
In [7]:
df.drop("Surname", axis=1, inplace=True)
In [ ]:
df.nunique()
Out[ ]:
0
CreditScore 460
Geography 3
Gender 2
Age 70
Tenure 11
Balance 6382
NumOfProducts 4
HasCrCard 2
IsActiveMember 2
EstimatedSalary 9999
Exited 2

In [ ]:
#run a for loop that looks for all the uniqe value of specified datatypes
sanity_check = df.select_dtypes(include=['int64','object']).columns
for var in sanity_check:
    print(var)
    print(df[var].unique())
    print()
CreditScore
[619 608 502 699 850 645 822 376 501 684 528 497 476 549 635 616 653 587
 726 732 636 510 669 846 577 756 571 574 411 591 533 553 520 722 475 490
 804 582 472 465 556 834 660 776 829 637 550 698 585 788 655 601 656 725
 511 614 742 687 555 603 751 581 735 661 675 738 813 657 604 519 664 678
 757 416 665 777 543 506 493 652 750 729 646 647 808 524 769 730 515 773
 814 710 413 623 670 622 785 605 479 685 538 562 721 628 668 828 674 625
 432 770 758 795 686 789 589 461 584 579 663 682 793 691 485 650 754 535
 716 539 706 586 631 717 800 683 704 615 667 484 480 578 512 606 597 778
 514 525 715 580 807 521 759 516 711 618 643 671 689 620 676 572 695 592
 567 694 547 594 673 610 767 763 712 703 662 659 523 772 545 634 739 771
 681 544 696 766 727 693 557 531 498 651 791 733 811 707 714 782 775 799
 602 744 588 747 583 627 731 629 438 642 806 474 559 429 680 749 734 644
 626 649 805 718 840 630 654 762 568 613 522 737 648 443 640 540 460 593
 801 611 802 745 483 690 492 709 705 560 752 701 537 487 596 702 486 724
 548 464 790 534 748 494 590 468 509 818 816 536 753 774 621 569 658 798
 641 542 692 639 765 570 638 599 632 779 527 564 833 504 842 508 417 598
 741 607 761 848 546 439 755 760 526 713 700 666 566 495 688 612 477 427
 839 819 720 459 503 624 529 563 482 796 445 746 786 554 672 787 499 844
 450 815 838 803 736 633 600 679 517 792 743 488 421 841 708 507 505 456
 435 561 518 565 728 784 552 609 764 697 723 551 444 719 496 541 830 812
 677 420 595 617 809 500 826 434 513 478 797 363 399 463 780 452 575 837
 794 824 428 823 781 849 489 431 457 768 831 359 820 573 576 558 817 449
 440 415 821 530 350 446 425 740 481 783 358 845 451 458 469 423 404 836
 473 835 466 491 351 827 843 365 532 414 453 471 401 810 832 470 447 422
 825 430 436 426 408 847 418 437 410 454 407 455 462 386 405 383 395 467
 433 442 424 448 441 367 412 382 373 419]

Geography
['France' 'Spain' 'Germany']

Gender
['Female' 'Male']

Age
[42 41 39 43 44 50 29 27 31 24 34 25 35 45 58 32 38 46 36 33 40 51 61 49
 37 19 66 56 26 21 55 75 22 30 28 65 48 52 57 73 47 54 72 20 67 79 62 53
 80 59 68 23 60 70 63 64 18 82 69 74 71 76 77 88 85 84 78 81 92 83]

Tenure
[ 2  1  8  7  4  6  3 10  5  9  0]

NumOfProducts
[1 3 2 4]

HasCrCard
[1 0]

IsActiveMember
[1 0]

Exited
[1 0]

In [ ]:
#for loop that gives a count of unique values for desired data type
for i in df.describe(include=["object"]).columns:
    print("Unique values in", i, "are :")
    print(df[i].value_counts())
    print("*" * 50)
Unique values in Geography are :
Geography
France     5014
Germany    2509
Spain      2477
Name: count, dtype: int64
**************************************************
Unique values in Gender are :
Gender
Male      5457
Female    4543
Name: count, dtype: int64
**************************************************
In [ ]:
#for loop that gives a count of unique values for desired data type
for i in df.describe(include=["int64"]).columns:
    print("Unique values in", i, "are :")
    print(df[i].value_counts())
    print("*" * 50)
Unique values in CreditScore are :
CreditScore
850    233
678     63
655     54
705     53
667     53
      ... 
404      1
351      1
365      1
417      1
419      1
Name: count, Length: 460, dtype: int64
**************************************************
Unique values in Age are :
Age
37    478
38    477
35    474
36    456
34    447
     ... 
92      2
82      1
88      1
85      1
83      1
Name: count, Length: 70, dtype: int64
**************************************************
Unique values in Tenure are :
Tenure
2     1048
1     1035
7     1028
8     1025
5     1012
3     1009
4      989
9      984
6      967
10     490
0      413
Name: count, dtype: int64
**************************************************
Unique values in NumOfProducts are :
NumOfProducts
1    5084
2    4590
3     266
4      60
Name: count, dtype: int64
**************************************************
Unique values in HasCrCard are :
HasCrCard
1    7055
0    2945
Name: count, dtype: int64
**************************************************
Unique values in IsActiveMember are :
IsActiveMember
1    5151
0    4849
Name: count, dtype: int64
**************************************************
Unique values in Exited are :
Exited
0    7963
1    2037
Name: count, dtype: int64
**************************************************
In [ ]:
for i in df.describe(include=["float64"]).columns:
    print("Unique values in", i, "are :")
    print(df[i].value_counts())
    print("*" * 50)
Unique values in Balance are :
Balance
0.00         3617
130170.82       2
105473.74       2
85304.27        1
159397.75       1
             ... 
81556.89        1
112687.69       1
108698.96       1
238387.56       1
130142.79       1
Name: count, Length: 6382, dtype: int64
**************************************************
Unique values in EstimatedSalary are :
EstimatedSalary
24924.92     2
101348.88    1
55313.44     1
72500.68     1
182692.80    1
            ..
120893.07    1
188377.21    1
55902.93     1
4523.74      1
38190.78     1
Name: count, Length: 9999, dtype: int64
**************************************************

Exploratory Data Analysis¶

Univariate Analysis¶

In [8]:
# function to plot a boxplot and a histogram along the same scale.


def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (12,7))
    kde: whether to the show density curve (default False)
    bins: number of bins for histogram (default None)
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,  # Number of rows of the subplot grid= 2
        sharex=True,  # x-axis will be shared among all subplots
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  # creating the 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  # boxplot will be created and a triangle will indicate the mean value of the column
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )  # For histogram
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    )  # Add mean to the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Add median to the histogram
In [9]:
# function to create labeled barplots


def labeled_barplot(data, feature, perc=False, n=None):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))

    plt.xticks(rotation=90, fontsize=15)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n].sort_values(),
    )

    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            )  # percentage of each class of the category
        else:
            label = p.get_height()  # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height()  # height of the plot

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage

    plt.show()  # show the plot
In [10]:
# function to plot stacked bar chart

def stacked_barplot(data, predictor, target):
    """
    Print the category counts and plot a stacked bar chart

    data: dataframe
    predictor: independent variable
    target: target variable
    """
    count = data[predictor].nunique()
    sorter = data[target].value_counts().index[-1]
    tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
        by=sorter, ascending=False
    )
    print(tab1)
    print("-" * 120)
    tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
        by=sorter, ascending=False
    )
    tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
    plt.legend(
        loc="lower left", frameon=False,
    )
    plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
    plt.show()
In [11]:
### Function to plot distributions

def distribution_plot_wrt_target(data, predictor, target):

    fig, axs = plt.subplots(2, 2, figsize=(12, 10))

    target_uniq = data[target].unique()

    axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
    sns.histplot(
        data=data[data[target] == target_uniq[0]],
        x=predictor,
        kde=True,
        ax=axs[0, 0],
        color="teal",
    )

    axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
    sns.histplot(
        data=data[data[target] == target_uniq[1]],
        x=predictor,
        kde=True,
        ax=axs[0, 1],
        color="orange",
    )

    axs[1, 0].set_title("Boxplot w.r.t target")
    sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")

    axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
    sns.boxplot(
        data=data,
        x=target,
        y=predictor,
        ax=axs[1, 1],
        showfliers=False,
        palette="gist_rainbow",
    )

    plt.tight_layout()
    plt.show()
In [119]:
def f1_minority(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))

    precision = true_positives / (predicted_positives + K.epsilon())
    recall = true_positives / (possible_positives + K.epsilon())

    f1_val = 2 * ((precision * recall) / (precision + recall + K.epsilon()))
    return f1_val
In [121]:
def f1_minority(y_true, y_pred):
    y_true = tf.cast(y_true, tf.float32)  # Cast y_true to float32
    y_pred = tf.cast(y_pred, tf.float32)  # Cast y_pred to float32

    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))

    precision = true_positives / (predicted_positives + K.epsilon())
    recall = true_positives / (possible_positives + K.epsilon())

    f1_val = 2 * ((precision * recall) / (precision + recall + K.epsilon()))
    return f1_val
In [123]:
def f1_minority(y_true, y_pred):
    y_true = tf.cast(y_true, tf.float32)
    y_pred = tf.cast(y_pred, tf.float32)

    tp = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    pp = K.sum(K.round(K.clip(y_pred, 0, 1)))
    fp = pp - tp
    fn = K.sum(K.round(K.clip(y_true, 0, 1))) - tp

    precision = tp / (tp + fp + K.epsilon())
    recall = tp / (tp + fn + K.epsilon())

    f1_val = 2 * ((precision * recall) / (precision + recall + K.epsilon()))
    return f1_val
In [ ]:
labeled_barplot(df, 'Exited')
In [ ]:
labeled_barplot(df, 'Geography')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  ax = sns.countplot(
No description has been provided for this image
In [ ]:
labeled_barplot(df, 'Gender')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  ax = sns.countplot(
No description has been provided for this image
In [ ]:
labeled_barplot(df, 'NumOfProducts')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  ax = sns.countplot(
No description has been provided for this image
In [ ]:
labeled_barplot(df, 'HasCrCard')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  ax = sns.countplot(
No description has been provided for this image
In [ ]:
labeled_barplot(df, 'IsActiveMember')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  ax = sns.countplot(
No description has been provided for this image
In [ ]:
labeled_barplot(df, 'Age')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  ax = sns.countplot(
No description has been provided for this image
In [ ]:
histogram_boxplot(df,'Age', kde=True)
No description has been provided for this image
In [ ]:
labeled_barplot(df, 'Tenure')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  ax = sns.countplot(
No description has been provided for this image
In [ ]:
histogram_boxplot(df,'Tenure', kde=True)
No description has been provided for this image
In [ ]:
histogram_boxplot(df,'CreditScore', kde=True)
No description has been provided for this image
In [ ]:
histogram_boxplot(df,'Balance', kde=True)
No description has been provided for this image
In [ ]:
histogram_boxplot(df,'EstimatedSalary', kde=True)
No description has been provided for this image

Bivariate Analysis¶

In [ ]:
sns.pairplot(df, hue='Exited')
Out[ ]:
<seaborn.axisgrid.PairGrid at 0x7a5a7072ecd0>
No description has been provided for this image
In [ ]:
plt.figure(figsize=(15, 7))
sns.heatmap(data.corr(numeric_only = True), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="coolwarm")
plt.show()
No description has been provided for this image

Two of our x's corolate with our y somehat already thats good

In [ ]:
pd.crosstab(df["Age"], df["Exited"]).plot(kind="bar")
plt.show()
No description has been provided for this image
In [ ]:
stacked_barplot(df,"Age","Exited")
Exited     0     1    All
Age                      
All     7963  2037  10000
46       135    91    226
40       343    89    432
43       209    88    297
45       142    87    229
..       ...   ...    ...
79         4     0      4
78         5     0      5
77        10     0     10
76        11     0     11
75         9     0      9

[71 rows x 3 columns]
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
distribution_plot_wrt_target(df,"Age","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
<ipython-input-183-3732c5f6b024>:31: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(
No description has been provided for this image
In [ ]:
pd.crosstab(df["Gender"], df["Exited"]).plot(kind="bar")
plt.show()
No description has been provided for this image
In [ ]:
stacked_barplot(df,"Gender","Exited")
Exited     0     1    All
Gender                   
All     7963  2037  10000
Female  3404  1139   4543
Male    4559   898   5457
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
pd.crosstab(df["Geography"], df["Exited"]).plot(kind="bar")
plt.show()
No description has been provided for this image
In [ ]:
stacked_barplot(df,"Geography","Exited")
Exited        0     1    All
Geography                   
All        7963  2037  10000
Germany    1695   814   2509
France     4204   810   5014
Spain      2064   413   2477
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
pd.crosstab(df["NumOfProducts"], df["Exited"]).plot(kind="bar")
plt.show()
No description has been provided for this image
In [ ]:
stacked_barplot(df,"NumOfProducts","Exited")
Exited            0     1    All
NumOfProducts                   
All            7963  2037  10000
1              3675  1409   5084
2              4242   348   4590
3                46   220    266
4                 0    60     60
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
distribution_plot_wrt_target(df,"NumOfProducts","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
<ipython-input-183-3732c5f6b024>:31: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(
No description has been provided for this image
In [ ]:
pd.crosstab(df["HasCrCard"], df["Exited"]).plot(kind="bar")
plt.show()
No description has been provided for this image
In [ ]:
stacked_barplot(df,"HasCrCard","Exited")
Exited        0     1    All
HasCrCard                   
All        7963  2037  10000
1          5631  1424   7055
0          2332   613   2945
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
pd.crosstab(df["IsActiveMember"], df["Exited"]).plot(kind="bar")
plt.show()
No description has been provided for this image
In [ ]:
stacked_barplot(df,"IsActiveMember","Exited")
Exited             0     1    All
IsActiveMember                   
All             7963  2037  10000
0               3547  1302   4849
1               4416   735   5151
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
pd.crosstab(df["Tenure"], df["Exited"]).plot(kind="bar")
plt.show()
No description has been provided for this image
In [ ]:
stacked_barplot(df,"Tenure","Exited")
Exited     0     1    All
Tenure                   
All     7963  2037  10000
1        803   232   1035
3        796   213   1009
9        771   213    984
5        803   209   1012
4        786   203    989
2        847   201   1048
8        828   197   1025
6        771   196    967
7        851   177   1028
10       389   101    490
0        318    95    413
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
distribution_plot_wrt_target(df,"Tenure","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
<ipython-input-183-3732c5f6b024>:31: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(
No description has been provided for this image
In [ ]:
distribution_plot_wrt_target(df,"CreditScore","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
<ipython-input-183-3732c5f6b024>:31: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(
No description has been provided for this image
In [ ]:
pd.crosstab(df["CreditScore"], df["Exited"]).plot(kind="bar")
plt.show()
No description has been provided for this image
In [ ]:
stacked_barplot(df,"CreditScore","Exited")
Exited          0     1    All
CreditScore                   
All          7963  2037  10000
850           190    43    233
651            33    17     50
705            37    16     53
637            32    14     46
...           ...   ...    ...
810             6     0      6
442             1     0      1
488            10     0     10
814            11     0     11
472             8     0      8

[461 rows x 3 columns]
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
distribution_plot_wrt_target(df,"Balance","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
<ipython-input-183-3732c5f6b024>:31: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(
No description has been provided for this image
In [ ]:
stacked_barplot(df,"Balance","Exited")
Exited        0     1    All
Balance                     
All        7963  2037  10000
0.0        3117   500   3617
12459.19      0     1      1
116973.26     0     1      1
116755.5      0     1      1
...         ...   ...    ...
117864.85     1     0      1
117852.26     1     0      1
85982.07      1     0      1
85996.19      1     0      1
127146.68     1     0      1

[6383 rows x 3 columns]
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image
In [ ]:
distribution_plot_wrt_target(df,"EstimatedSalary","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
<ipython-input-183-3732c5f6b024>:31: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(
No description has been provided for this image
In [ ]:
stacked_barplot(df,"EstimatedSalary","Exited")
Exited              0     1    All
EstimatedSalary                   
All              7963  2037  10000
93132.61            0     1      1
91167.19            0     1      1
91556.57            0     1      1
91560.63            0     1      1
...               ...   ...    ...
23237.25            1     0      1
90384.26            1     0      1
90346.1             1     0      1
90305.97            1     0      1
100236.02           1     0      1

[10000 rows x 3 columns]
------------------------------------------------------------------------------------------------------------------------
No description has been provided for this image

Data Preprocessing¶

Dummy variables to prevent leakage

In [8]:
df = pd.get_dummies(df,columns=df.select_dtypes(include=["object"]).columns.tolist(),drop_first=True)
df = df.astype(float)
df.head()
Out[8]:
CreditScore Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited Geography_Germany Geography_Spain Gender_Male
0 619.0 42.0 2.0 0.00 1.0 1.0 1.0 101348.88 1.0 0.0 0.0 0.0
1 608.0 41.0 1.0 83807.86 1.0 0.0 1.0 112542.58 0.0 0.0 1.0 0.0
2 502.0 42.0 8.0 159660.80 3.0 1.0 0.0 113931.57 1.0 0.0 0.0 0.0
3 699.0 39.0 1.0 0.00 2.0 0.0 0.0 93826.63 0.0 0.0 0.0 0.0
4 850.0 43.0 2.0 125510.82 1.0 1.0 1.0 79084.10 0.0 0.0 1.0 0.0

Train-validation-test Split¶

In [9]:
## Storing required categorical variables to apply dummification
cols_list = ['CreditScore', 'Age', 'Tenure', 'Balance', 'NumOfProducts', 'EstimatedSalary']
In [10]:
X = df.drop(['Exited'],axis=1)
y = df['Exited']
print(X.shape)
print(y.shape)
print(type(X))
(10000, 11)
(10000,)
<class 'pandas.core.frame.DataFrame'>
In [11]:
# Splitting the dataset into the Training and Testing set.

X_large, X_test, y_large, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1,stratify=y,shuffle = True)
X_train, X_val, y_train, y_val = train_test_split(X_large, y_large, test_size = 0.2, random_state = 1,stratify=y_large, shuffle = True)
In [12]:
print(X_train.shape, X_val.shape, X_test.shape)
(6400, 11) (1600, 11) (2000, 11)
In [13]:
print(y_train.shape, y_val.shape, y_test.shape)
(6400,) (1600,) (2000,)

Data Normalization¶

In [14]:
# creating an instance of the standard scaler
sc = StandardScaler()

X_train[cols_list] = sc.fit_transform(X_train[cols_list])
X_val[cols_list] = sc.transform(X_val[cols_list])
X_test[cols_list] = sc.transform(X_test[cols_list])

i will only do batch normalization on some models and layers but not yet

Model Building¶

Model Evaluation Criterion¶

In [15]:
# defining a function to compute different metrics to check performance of a classification model built using sklearn
def model_performance_classification_sklearn(model, predictors, target):
    """
    Function to compute different metrics to check classification model performance

    model: classifier
    predictors: independent variables
    target: dependent variable
    """

    # predicting using the independent variables
    pred = model.predict(predictors)

    acc = accuracy_score(target, pred)  # to compute Accuracy
    recall = recall_score(target, pred)  # to compute Recall
    precision = precision_score(target, pred)  # to compute Precision
    f1 = f1_score(target, pred)  # to compute F1-score

    # creating a dataframe of metrics
    df_perf = pd.DataFrame(
        {
            "Accuracy": acc,
            "Recall": recall,
            "Precision": precision,
            "F1": f1

        },
        index=[0],
    )

    return df_perf
In [16]:
def plot(history, name):
    """
    Function to plot loss/accuracy

    history: an object which stores the metrics and losses.
    name: can be one of Loss or Accuracy
    """
    fig, ax = plt.subplots() #Creating a subplot with figure and axes.
    plt.plot(history.history[name]) #Plotting the train accuracy or train loss
    plt.plot(history.history['val_'+name]) #Plotting the validation accuracy or validation loss

    plt.title('Model ' + name.capitalize()) #Defining the title of the plot.
    plt.ylabel(name.capitalize()) #Capitalizing the first letter.
    plt.xlabel('Epoch') #Defining the label for the x-axis.
    fig.legend(['Train', 'Validation'], loc="outside right upper") #Defining the legend, loc controls the position of the legend.
In [17]:
# Defining the columns of the dataframe which are nothing but the hyper parameters and the metrics.
columns = ["# hidden layers", "# neurons - hidden layer", "activation function - hidden layer", "# epochs", "batch size", "optimizer", "learning rate", "initializer", "regularizer", "train loss", "validation loss", "train recall", "validation recall", "time (secs)"]

# Creating a pandas dataframe.
results = pd.DataFrame(columns=columns)

Write down the logic for choosing the metric that would be the best metric for this business scenario.

Recall is the preferred metric because we want to minimize false negatives, ensuring that as many potential churners as possible are identified. we want to make personalized offers or some kind of outreach. The bottom line wont hurt too much if we send out a few extra offers.

Neural Network with SGD Optimizer¶

In [18]:
tf.keras.backend.clear_session()

model_0 = Sequential()
model_0.add(Dense(128, activation='relu', input_dim=X_train.shape[1]))
model_0.add(Dense(64, activation='relu'))
model_0.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001)
model_0.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

start = time.time()
history = model_0.fit(X_train, y_train, validation_data=(X_val, y_val), batch_size=16, epochs=10)
end = time.time()
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Epoch 1/10
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
399/400 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step - loss: 0.6929 - recall: 0.6014
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

400/400 ━━━━━━━━━━━━━━━━━━━━ 16s 36ms/step - loss: 0.6927 - recall: 0.6001 - val_loss: 0.5907 - val_recall: 0.0031
Epoch 2/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 36ms/step - loss: 0.5748 - recall: 0.0029 - val_loss: 0.5326 - val_recall: 0.0000e+00
Epoch 3/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 36ms/step - loss: 0.5229 - recall: 0.0000e+00 - val_loss: 0.5067 - val_recall: 0.0000e+00
Epoch 4/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.5002 - recall: 0.0000e+00 - val_loss: 0.4926 - val_recall: 0.0000e+00
Epoch 5/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 36ms/step - loss: 0.4874 - recall: 0.0000e+00 - val_loss: 0.4834 - val_recall: 0.0000e+00
Epoch 6/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.4857 - recall: 0.0000e+00 - val_loss: 0.4763 - val_recall: 0.0000e+00
Epoch 7/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.4822 - recall: 0.0000e+00 - val_loss: 0.4703 - val_recall: 0.0000e+00
Epoch 8/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 36ms/step - loss: 0.4767 - recall: 0.0000e+00 - val_loss: 0.4650 - val_recall: 0.0000e+00
Epoch 9/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.4605 - recall: 0.0000e+00 - val_loss: 0.4602 - val_recall: 0.0000e+00
Epoch 10/10
400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.4683 - recall: 0.0000e+00 - val_loss: 0.4558 - val_recall: 0.0031
In [19]:
print("Time taken in seconds ",end-start)
Time taken in seconds  143.8668487071991
In [20]:
plot(history,'loss')
No description has been provided for this image

it looks like 10 epochs is not enough its possoible the momentum is pushing it out of helpful minima

In [21]:
plot(history,'recall')
No description has been provided for this image
In [22]:
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt

# Generate predictions on the validation set using model_0
val_pred = model_0.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
38/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
              precision    recall  f1-score   support

         0.0       0.80      1.00      0.89      1274
         1.0       1.00      0.00      0.01       326

    accuracy                           0.80      1600
   macro avg       0.90      0.50      0.45      1600
weighted avg       0.84      0.80      0.71      1600

No description has been provided for this image

here there is a major class imbalance which is severely impacting model performance when it comes to recall of our target variable. not sure whats going on with the heat map

In [23]:
results.loc[0] = [2, "128,64", "relu,relu", 10, 16, "sgd", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [24]:
results
Out[24]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.0 0.003067 143.87

Model Performance Improvement¶

Neural Network with Adam Optimizer¶

In [36]:
tf.keras.backend.clear_session()

model_1 = Sequential()

# Get the correct input shape from your scaled training data
input_shape = X_train.shape[1]

# Input layer with 64 neurons and ReLU activation
model_1.add(Dense(64, activation='relu', input_shape=(input_shape,)))  # Use input_shape

# Hidden layer with 32 neurons and ReLU activation
model_1.add(Dense(32, activation='relu'))

# Output layer with 1 neuron and sigmoid activation
model_1.add(Dense(1, activation='sigmoid'))

# Compile the model
optimizer = keras.optimizers.Adam(clipnorm=1.0)
model_1.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

# Train the model (adjust epochs and batch size as needed)
history = model_1.fit(X_train, y_train, validation_data=(X_val, y_val),
                            batch_size=16, epochs=50)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 0s 60ms/step - loss: 0.4753 - recall: 0.0955
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

400/400 ━━━━━━━━━━━━━━━━━━━━ 27s 65ms/step - loss: 0.4751 - recall: 0.0957 - val_loss: 0.3744 - val_recall: 0.3313
Epoch 2/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 27s 67ms/step - loss: 0.3817 - recall: 0.3819 - val_loss: 0.3568 - val_recall: 0.4387
Epoch 3/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3547 - recall: 0.4532 - val_loss: 0.3584 - val_recall: 0.4693
Epoch 4/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.3382 - recall: 0.4808 - val_loss: 0.3518 - val_recall: 0.4663
Epoch 5/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.3319 - recall: 0.4711 - val_loss: 0.3500 - val_recall: 0.4755
Epoch 6/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3318 - recall: 0.5027 - val_loss: 0.3492 - val_recall: 0.4417
Epoch 7/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3240 - recall: 0.4521 - val_loss: 0.3487 - val_recall: 0.4264
Epoch 8/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3225 - recall: 0.4859 - val_loss: 0.3492 - val_recall: 0.5000
Epoch 9/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.3222 - recall: 0.5052 - val_loss: 0.3514 - val_recall: 0.4294
Epoch 10/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3255 - recall: 0.5173 - val_loss: 0.3481 - val_recall: 0.5245
Epoch 11/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.3149 - recall: 0.5226 - val_loss: 0.3500 - val_recall: 0.4417
Epoch 12/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3178 - recall: 0.4986 - val_loss: 0.3510 - val_recall: 0.4356
Epoch 13/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.2992 - recall: 0.5288 - val_loss: 0.3530 - val_recall: 0.4969
Epoch 14/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.3210 - recall: 0.4959 - val_loss: 0.3723 - val_recall: 0.3497
Epoch 15/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3249 - recall: 0.5259 - val_loss: 0.3643 - val_recall: 0.3834
Epoch 16/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3098 - recall: 0.5158 - val_loss: 0.3490 - val_recall: 0.4755
Epoch 17/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3114 - recall: 0.5318 - val_loss: 0.3550 - val_recall: 0.4080
Epoch 18/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3202 - recall: 0.5158 - val_loss: 0.3562 - val_recall: 0.4264
Epoch 19/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2973 - recall: 0.5487 - val_loss: 0.3615 - val_recall: 0.4233
Epoch 20/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.3043 - recall: 0.5353 - val_loss: 0.3534 - val_recall: 0.4724
Epoch 21/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2971 - recall: 0.5711 - val_loss: 0.3549 - val_recall: 0.4693
Epoch 22/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3036 - recall: 0.5502 - val_loss: 0.3570 - val_recall: 0.4479
Epoch 23/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3027 - recall: 0.5286 - val_loss: 0.3562 - val_recall: 0.4939
Epoch 24/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2968 - recall: 0.5554 - val_loss: 0.3580 - val_recall: 0.4356
Epoch 25/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2885 - recall: 0.5471 - val_loss: 0.3584 - val_recall: 0.4601
Epoch 26/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2933 - recall: 0.5595 - val_loss: 0.3600 - val_recall: 0.4479
Epoch 27/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 27s 66ms/step - loss: 0.2843 - recall: 0.5835 - val_loss: 0.3571 - val_recall: 0.4632
Epoch 28/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2783 - recall: 0.5796 - val_loss: 0.3624 - val_recall: 0.4509
Epoch 29/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2821 - recall: 0.5689 - val_loss: 0.3608 - val_recall: 0.4417
Epoch 30/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2880 - recall: 0.5597 - val_loss: 0.3639 - val_recall: 0.4509
Epoch 31/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2758 - recall: 0.5944 - val_loss: 0.3724 - val_recall: 0.4724
Epoch 32/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2913 - recall: 0.5586 - val_loss: 0.3665 - val_recall: 0.5307
Epoch 33/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2723 - recall: 0.5827 - val_loss: 0.3621 - val_recall: 0.5460
Epoch 34/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.2826 - recall: 0.5878 - val_loss: 0.3630 - val_recall: 0.4724
Epoch 35/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2680 - recall: 0.5896 - val_loss: 0.3672 - val_recall: 0.4969
Epoch 36/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2780 - recall: 0.5743 - val_loss: 0.3727 - val_recall: 0.5031
Epoch 37/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 27s 66ms/step - loss: 0.2689 - recall: 0.5918 - val_loss: 0.3679 - val_recall: 0.4816
Epoch 38/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2730 - recall: 0.5957 - val_loss: 0.3770 - val_recall: 0.3957
Epoch 39/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2761 - recall: 0.5741 - val_loss: 0.3675 - val_recall: 0.4693
Epoch 40/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2698 - recall: 0.5916 - val_loss: 0.3697 - val_recall: 0.4693
Epoch 41/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2689 - recall: 0.6139 - val_loss: 0.3755 - val_recall: 0.4755
Epoch 42/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2538 - recall: 0.6080 - val_loss: 0.3721 - val_recall: 0.5337
Epoch 43/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.2725 - recall: 0.5899 - val_loss: 0.3834 - val_recall: 0.5460
Epoch 44/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2719 - recall: 0.6008 - val_loss: 0.3764 - val_recall: 0.4202
Epoch 45/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2619 - recall: 0.5966 - val_loss: 0.3861 - val_recall: 0.5092
Epoch 46/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2633 - recall: 0.6177 - val_loss: 0.3806 - val_recall: 0.4325
Epoch 47/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2642 - recall: 0.5845 - val_loss: 0.3748 - val_recall: 0.4693
Epoch 48/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2705 - recall: 0.5773 - val_loss: 0.3801 - val_recall: 0.4663
Epoch 49/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2659 - recall: 0.6079 - val_loss: 0.3870 - val_recall: 0.4816
Epoch 50/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2452 - recall: 0.6384 - val_loss: 0.3858 - val_recall: 0.4571
In [37]:
print("Time taken in seconds ",end-start)
Time taken in seconds  606.9027745723724
In [38]:
plot(history,'loss')
No description has been provided for this image

better but more overfitting

In [39]:
plot(history,'recall')
No description has been provided for this image

more oscilations but better performance

In [40]:
val_pred = model_1.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
40/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
              precision    recall  f1-score   support

         0.0       0.87      0.95      0.91      1274
         1.0       0.70      0.46      0.55       326

    accuracy                           0.85      1600
   macro avg       0.78      0.70      0.73      1600
weighted avg       0.84      0.85      0.84      1600

No description has been provided for this image

adam does better with the same batch size and number of epochs even with less complexity in term,s of parameters! at least on this dataset with class imbalances of this kind and of this size and complexity.

In [41]:
results.loc[1] = [2, "64,32", "relu,relu", 50, 16, "Adam", "0, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [42]:
results
Out[42]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90

we need to make sure we dont negelect precision at the expense of recall too much so we need to try both recall and accuracy and f1 score as our metric

In [43]:
tf.keras.backend.clear_session()

model_2 = Sequential()
model_2.add(Dense(128, activation="relu", input_dim=X_train.shape[1]))
model_2.add(BatchNormalization())  # Batch Normalization layer
model_2.add(Dense(64, activation="relu"))
model_2.add(BatchNormalization())  # Batch Normalization layer
model_2.add(Dense(32, activation='relu'))
model_2.add(BatchNormalization())  # Batch Normalization layer
model_2.add(Dense(1, activation='relu'))

optimizer = keras.optimizers.Adam(clipnorm=1.0)  # Adam with gradient clipping
model_2.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy']) #trying accuracy

early_stopping = EarlyStopping(monitor='val_accuracy', patience=5, restore_best_weights=True) #stops if accuracy doenst improve after 5 epochs

start = time.time()
history = model_2.fit(X_train, y_train, validation_data=(X_val, y_val),
                    batch_size=100, epochs=25, callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/25
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 118ms/step - accuracy: 0.6761 - loss: 3.5233
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

64/64 ━━━━━━━━━━━━━━━━━━━━ 9s 125ms/step - accuracy: 0.6769 - loss: 3.5130 - val_accuracy: 0.8006 - val_loss: 2.9040
Epoch 2/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 123ms/step - accuracy: 0.8175 - loss: 1.8314 - val_accuracy: 0.8000 - val_loss: 2.7212
Epoch 3/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 123ms/step - accuracy: 0.8362 - loss: 1.5584 - val_accuracy: 0.8106 - val_loss: 1.9331
Epoch 4/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 124ms/step - accuracy: 0.8512 - loss: 1.2954 - val_accuracy: 0.8238 - val_loss: 1.6080
Epoch 5/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8605 - loss: 1.1375 - val_accuracy: 0.8219 - val_loss: 1.1258
Epoch 6/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 124ms/step - accuracy: 0.8554 - loss: 1.0733 - val_accuracy: 0.8256 - val_loss: 1.2252
Epoch 7/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 123ms/step - accuracy: 0.8646 - loss: 0.9337 - val_accuracy: 0.8400 - val_loss: 1.0676
Epoch 8/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8681 - loss: 0.8825 - val_accuracy: 0.8481 - val_loss: 1.1021
Epoch 9/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 124ms/step - accuracy: 0.8667 - loss: 0.8746 - val_accuracy: 0.8394 - val_loss: 0.9627
Epoch 10/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8699 - loss: 0.7232 - val_accuracy: 0.8350 - val_loss: 1.0835
Epoch 11/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8675 - loss: 0.6428 - val_accuracy: 0.8250 - val_loss: 0.8817
Epoch 12/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8681 - loss: 0.6248 - val_accuracy: 0.8444 - val_loss: 0.8359
Epoch 13/25
64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 125ms/step - accuracy: 0.8815 - loss: 0.5425 - val_accuracy: 0.8344 - val_loss: 0.8924
Time taken in seconds  103.79582643508911

now we will tune this adam one

In [44]:
plot(history,'loss')
No description has been provided for this image
In [45]:
plot(history,'accuracy')
No description has been provided for this image

recall is actually worse with more parameters here

In [46]:
val_pred = model_2.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
13/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step 
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step
              precision    recall  f1-score   support

         0.0       0.87      0.95      0.91      1274
         1.0       0.69      0.46      0.55       326

    accuracy                           0.85      1600
   macro avg       0.78      0.70      0.73      1600
weighted avg       0.84      0.85      0.84      1600

No description has been provided for this image

the model is paying too much attention to the majority class but its getting a decent balance between recall and precision on our target variable the model needs more tuning for sure

the class imbalance is still an issue despite the added and tuned parameters.

The class imbalance is influencing the model's learning process, causing it to prioritize the majority class even with tuned parameters and adaptive learning rates.

SGD with momentum performs better in scenarios with imbalanced datasets in many cases so we will try that first.

Neural Network with Balanced Data (by applying SMOTE) and SGD Optimizer¶

In [26]:
sm  = SMOTE(random_state=1)
X_train_smote, y_train_smote= sm.fit_resample(X_train, y_train)
print('After UpSampling, the shape of train_X: {}'.format(X_train_smote.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train_smote.shape))
After UpSampling, the shape of train_X: (10192, 11)
After UpSampling, the shape of train_y: (10192,) 

In [27]:
# Scale the SMOTE-resampled data
scaler_smote = StandardScaler()  # Create a new scaler for SMOTE data
X_train_smote_scaled = scaler_smote.fit_transform(X_train_smote)
In [47]:
tf.keras.backend.clear_session()

model_3 = Sequential()
model_3.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1]))
model_3.add(Dense(64, activation="relu"))
model_3.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_3.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

early_stopping = EarlyStopping(monitor='val_recall', patience=5, restore_best_weights=True)

start = time.time()
history = model_3.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    batch_size=32, epochs=100, callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.6750 - recall: 0.6456
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6749 - recall: 0.6457 - val_loss: 0.6121 - val_recall: 0.6442
Epoch 2/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5953 - recall: 0.7106 - val_loss: 0.5602 - val_recall: 0.6748
Epoch 3/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5564 - recall: 0.7139 - val_loss: 0.5393 - val_recall: 0.6963
Epoch 4/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5353 - recall: 0.7193 - val_loss: 0.5150 - val_recall: 0.6994
Epoch 5/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5152 - recall: 0.7187 - val_loss: 0.5251 - val_recall: 0.7607
Epoch 6/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4967 - recall: 0.7591 - val_loss: 0.5063 - val_recall: 0.7515
Epoch 7/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4818 - recall: 0.7617 - val_loss: 0.5162 - val_recall: 0.7791
Epoch 8/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4652 - recall: 0.7765 - val_loss: 0.5300 - val_recall: 0.7945
Epoch 9/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 13s 40ms/step - loss: 0.4530 - recall: 0.7935 - val_loss: 0.5347 - val_recall: 0.7914
Epoch 10/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 13s 39ms/step - loss: 0.4417 - recall: 0.7939 - val_loss: 0.5603 - val_recall: 0.8160
Epoch 11/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4355 - recall: 0.7979 - val_loss: 0.5510 - val_recall: 0.8098
Epoch 12/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4324 - recall: 0.7840 - val_loss: 0.5737 - val_recall: 0.8282
Epoch 13/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 13s 40ms/step - loss: 0.4237 - recall: 0.8001 - val_loss: 0.5697 - val_recall: 0.8129
Epoch 14/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4220 - recall: 0.7935 - val_loss: 0.5720 - val_recall: 0.8129
Epoch 15/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4201 - recall: 0.7958 - val_loss: 0.5881 - val_recall: 0.8252
Epoch 16/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4110 - recall: 0.8065 - val_loss: 0.6081 - val_recall: 0.8374
Epoch 17/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4143 - recall: 0.8028 - val_loss: 0.5953 - val_recall: 0.8344
Epoch 18/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4194 - recall: 0.7942 - val_loss: 0.5994 - val_recall: 0.8313
Epoch 19/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4056 - recall: 0.8047 - val_loss: 0.5826 - val_recall: 0.8190
Epoch 20/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4025 - recall: 0.7908 - val_loss: 0.6612 - val_recall: 0.8589
Epoch 21/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4061 - recall: 0.8095 - val_loss: 0.6422 - val_recall: 0.8344
Epoch 22/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4089 - recall: 0.8148 - val_loss: 0.6824 - val_recall: 0.8620
Epoch 23/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4017 - recall: 0.8150 - val_loss: 0.6773 - val_recall: 0.8620
Epoch 24/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4011 - recall: 0.8064 - val_loss: 0.6790 - val_recall: 0.8589
Epoch 25/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4002 - recall: 0.8009 - val_loss: 0.6869 - val_recall: 0.8589
Epoch 26/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.3918 - recall: 0.8169 - val_loss: 0.6989 - val_recall: 0.8558
Epoch 27/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.3812 - recall: 0.8175 - val_loss: 0.7176 - val_recall: 0.8589
Time taken in seconds  332.36059737205505
In [48]:
plot(history,'loss')
No description has been provided for this image

here the loss is still a major problem despite the smote fix we will have to do more tuning simply adding more epochs or adjusting the learning rates wont fix this...

In [49]:
plot(history,'recall')
No description has been provided for this image

this is much better but the oscilations are still a bit wild.... more epochs and a slower learning rate should be included in the next model. I think playing with the momentum may also be a good idea.

In [50]:
val_pred = model_3.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
40/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
              precision    recall  f1-score   support

         0.0       0.94      0.57      0.71      1274
         1.0       0.34      0.86      0.49       326

    accuracy                           0.63      1600
   macro avg       0.64      0.72      0.60      1600
weighted avg       0.82      0.63      0.67      1600

No description has been provided for this image

our risk management department may or may not enjoy this... we amy want to switch to f1 so we can gett better precisioon

In [51]:
results.loc[3] = [2, "128,64", "relu,relu", 100, 32, "SGD", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [52]:
results
Out[52]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36

Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer¶

In [53]:
model_4 = Sequential()
model_4.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1],
                 kernel_regularizer=regularizers.l2(0.01)))
model_4.add(BatchNormalization())
model_4.add(Dropout(0.5))  # Add dropout
model_4.add(Dense(64, activation="relu", kernel_regularizer=regularizers.l1(0.001)))
model_4.add(BatchNormalization())
model_4.add(Dropout(0.5))  # Add dropout
model_4.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.Adam()
model_4.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

early_stopping = EarlyStopping(monitor='val_recall', patience=5, restore_best_weights=True)

start = time.time()
history = model_4.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    batch_size=32, epochs=50,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 98ms/step - loss: 1.5942 - recall: 0.6297
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 1.5938 - recall: 0.6298 - val_loss: 1.2435 - val_recall: 0.7975
Epoch 2/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 1.1816 - recall: 0.7091 - val_loss: 1.0439 - val_recall: 0.8037
Epoch 3/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 0.9515 - recall: 0.7525 - val_loss: 0.9200 - val_recall: 0.8344
Epoch 4/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 105ms/step - loss: 0.8064 - recall: 0.7616 - val_loss: 0.8204 - val_recall: 0.8282
Epoch 5/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 0.7106 - recall: 0.7632 - val_loss: 0.7390 - val_recall: 0.8282
Epoch 6/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 0.6294 - recall: 0.7748 - val_loss: 0.6246 - val_recall: 0.7607
Epoch 7/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 0.6003 - recall: 0.7791 - val_loss: 0.6466 - val_recall: 0.7853
Epoch 8/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 102ms/step - loss: 0.5776 - recall: 0.7710 - val_loss: 0.6591 - val_recall: 0.7975
Time taken in seconds  264.18181252479553
In [54]:
plot(history,'loss')
No description has been provided for this image
In [55]:
plot(history,'recall')
No description has been provided for this image
In [56]:
val_pred = model_4.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
22/50 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
              precision    recall  f1-score   support

         0.0       0.94      0.64      0.76      1274
         1.0       0.37      0.83      0.51       326

    accuracy                           0.68      1600
   macro avg       0.65      0.74      0.64      1600
weighted avg       0.82      0.68      0.71      1600

No description has been provided for this image
In [57]:
results.loc[4] = [2, "128,64", "relu,relu", 100, 32, "Adam", "0, -", "xavier", "L2,L1", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [58]:
results
Out[58]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18

SGD does indeed do better on imbalanced data at least in this case...

Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout¶

In [59]:
model_5 = Sequential()
model_5.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1],
                 kernel_regularizer=regularizers.l2(0.01)))
model_5.add(BatchNormalization())
model_5.add(Dropout(0.5))
model_5.add(Dense(64, activation="relu", kernel_regularizer=regularizers.l1(0.001)))
model_5.add(BatchNormalization())
model_5.add(Dropout(0.5))
model_5.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.Adam()
model_5.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

early_stopping = EarlyStopping(monitor='val_recall', patience=5, restore_best_weights=True)

start = time.time()
history = model_5.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    batch_size=32, epochs=100,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 99ms/step - loss: 1.6560 - recall: 0.6191
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 1.6555 - recall: 0.6192 - val_loss: 1.3356 - val_recall: 0.8712
Epoch 2/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 1.2473 - recall: 0.7112 - val_loss: 1.1217 - val_recall: 0.8589
Epoch 3/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 1.0160 - recall: 0.7429 - val_loss: 0.9498 - val_recall: 0.8129
Epoch 4/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 0.8482 - recall: 0.7444 - val_loss: 0.7906 - val_recall: 0.7638
Epoch 5/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 0.7272 - recall: 0.7484 - val_loss: 0.7154 - val_recall: 0.8160
Epoch 6/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 0.6460 - recall: 0.7724 - val_loss: 0.6930 - val_recall: 0.8067
Time taken in seconds  198.04284501075745
In [60]:
plot(history,'loss')
No description has been provided for this image
In [61]:
plot(history,'recall')
No description has been provided for this image
In [62]:
val_pred = model_5.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
22/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
              precision    recall  f1-score   support

         0.0       0.95      0.59      0.72      1274
         1.0       0.35      0.87      0.50       326

    accuracy                           0.64      1600
   macro avg       0.65      0.73      0.61      1600
weighted avg       0.83      0.64      0.68      1600

No description has been provided for this image
In [63]:
results.loc[5] = [2, "128,64", "relu,relu", 100, 32, "Adam", "0, -", "xavier", "L2,L1", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [64]:
results
Out[64]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04

this did not go as expected so it might be a good idea to try a smaller batch size

In [65]:
model_6 = Sequential()

model_6.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1],
                 kernel_regularizer=regularizers.l2(0.01)))  # L2 regularization
model_6.add(BatchNormalization())
model_6.add(Dropout(0.5))
model_6.add(Dense(64, activation="relu",
                 kernel_regularizer=regularizers.l1(0.001)))  # L1 regularization
model_6.add(BatchNormalization())
model_6.add(Dropout(0.5))


model_6.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.Adam()
model_6.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall']) #recall

early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

start = time.time()
history = model_6.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    batch_size=20, epochs=50,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
510/510 ━━━━━━━━━━━━━━━━━━━━ 0s 99ms/step - loss: 1.6235 - recall: 0.6274
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 103ms/step - loss: 1.6232 - recall: 0.6275 - val_loss: 1.1850 - val_recall: 0.7914
Epoch 2/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 1.1569 - recall: 0.7094 - val_loss: 0.9852 - val_recall: 0.7945
Epoch 3/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 103ms/step - loss: 0.9041 - recall: 0.7510 - val_loss: 0.8212 - val_recall: 0.7914
Epoch 4/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 104ms/step - loss: 0.7563 - recall: 0.7525 - val_loss: 0.7445 - val_recall: 0.8067
Epoch 5/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 104ms/step - loss: 0.6671 - recall: 0.7568 - val_loss: 0.7064 - val_recall: 0.7975
Epoch 6/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.6168 - recall: 0.7636 - val_loss: 0.6678 - val_recall: 0.8098
Epoch 7/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.6013 - recall: 0.7584 - val_loss: 0.6084 - val_recall: 0.7730
Epoch 8/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.5857 - recall: 0.7566 - val_loss: 0.5993 - val_recall: 0.7577
Epoch 9/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 103ms/step - loss: 0.5750 - recall: 0.7612 - val_loss: 0.6284 - val_recall: 0.7822
Epoch 10/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.5627 - recall: 0.7661 - val_loss: 0.6188 - val_recall: 0.8221
Epoch 11/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.5582 - recall: 0.7689 - val_loss: 0.6378 - val_recall: 0.8098
Epoch 12/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 103ms/step - loss: 0.5611 - recall: 0.7763 - val_loss: 0.6252 - val_recall: 0.7975
Epoch 13/50
510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 103ms/step - loss: 0.5540 - recall: 0.7722 - val_loss: 0.6071 - val_recall: 0.7791
Time taken in seconds  681.8022449016571
In [66]:
plot(history,'loss')
No description has been provided for this image
In [67]:
plot(history,'recall')
No description has been provided for this image
In [68]:
val_pred = model_6.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
22/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
              precision    recall  f1-score   support

         0.0       0.92      0.74      0.82      1274
         1.0       0.43      0.76      0.55       326

    accuracy                           0.74      1600
   macro avg       0.68      0.75      0.69      1600
weighted avg       0.82      0.74      0.77      1600

No description has been provided for this image

accuracy is better and so is precison of our target variable... but recall is slipping

In [69]:
results.loc[6] = [2, "128,64", "relu,relu", 50, 20, "Adam", "0, -", "xavier", "L2,L1", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [70]:
results
Out[70]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04
6 2 128,64 relu,relu 50 8 Adam 0, - xavier L2,L1 0.550212 0.607145 0.777473 0.779141 681.80

now we will go back to SGD becuase that handles the imbalance best and tweak our best model to get better performance

In [71]:
tf.keras.backend.clear_session()

model_7 = Sequential()
model_7.add(Dense(64, activation="tanh", input_dim=X_train_smote.shape[1])) #tanh
model_7.add(Dense(32, activation="tanh"))#tanh
model_7.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_7.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

start = time.time()
history = model_7.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    batch_size=32, epochs=50,
                    callbacks=[early_stopping])
end = time.time()
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.6419 - recall: 0.6238
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6418 - recall: 0.6240 - val_loss: 0.5573 - val_recall: 0.6626
Epoch 2/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5602 - recall: 0.7059 - val_loss: 0.5554 - val_recall: 0.6656
Epoch 3/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.5611 - recall: 0.7009 - val_loss: 0.5577 - val_recall: 0.7086
Epoch 4/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5560 - recall: 0.7080 - val_loss: 0.5593 - val_recall: 0.7147
Epoch 5/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5471 - recall: 0.7161 - val_loss: 0.5561 - val_recall: 0.7117
Epoch 6/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5515 - recall: 0.7139 - val_loss: 0.5370 - val_recall: 0.7086
Epoch 7/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5470 - recall: 0.7069 - val_loss: 0.5439 - val_recall: 0.7301
Epoch 8/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5338 - recall: 0.7310 - val_loss: 0.5389 - val_recall: 0.7515
Epoch 9/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5208 - recall: 0.7322 - val_loss: 0.5365 - val_recall: 0.7515
Epoch 10/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5116 - recall: 0.7561 - val_loss: 0.5211 - val_recall: 0.7607
Epoch 11/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5010 - recall: 0.7454 - val_loss: 0.5223 - val_recall: 0.7638
Epoch 12/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4951 - recall: 0.7643 - val_loss: 0.5172 - val_recall: 0.7761
Epoch 13/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4827 - recall: 0.7750 - val_loss: 0.5109 - val_recall: 0.7730
Epoch 14/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4726 - recall: 0.7720 - val_loss: 0.5247 - val_recall: 0.7975
Epoch 15/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4592 - recall: 0.7852 - val_loss: 0.5116 - val_recall: 0.7730
Epoch 16/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4670 - recall: 0.7714 - val_loss: 0.4973 - val_recall: 0.7761
Epoch 17/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4560 - recall: 0.7743 - val_loss: 0.5017 - val_recall: 0.7638
Epoch 18/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4523 - recall: 0.7876 - val_loss: 0.5096 - val_recall: 0.7883
Epoch 19/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4520 - recall: 0.7760 - val_loss: 0.4970 - val_recall: 0.7730
Epoch 20/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4465 - recall: 0.7754 - val_loss: 0.5119 - val_recall: 0.7883
Epoch 21/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4426 - recall: 0.7831 - val_loss: 0.4983 - val_recall: 0.7730
Epoch 22/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4380 - recall: 0.7832 - val_loss: 0.5039 - val_recall: 0.7761
Epoch 23/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4282 - recall: 0.7984 - val_loss: 0.4962 - val_recall: 0.7699
Epoch 24/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4399 - recall: 0.7903 - val_loss: 0.5092 - val_recall: 0.7791
Epoch 25/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4222 - recall: 0.8007 - val_loss: 0.4948 - val_recall: 0.7546
Epoch 26/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4321 - recall: 0.7908 - val_loss: 0.4955 - val_recall: 0.7638
Epoch 27/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4200 - recall: 0.7906 - val_loss: 0.4961 - val_recall: 0.7577
Epoch 28/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4233 - recall: 0.8025 - val_loss: 0.4897 - val_recall: 0.7423
Epoch 29/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4226 - recall: 0.7962 - val_loss: 0.4897 - val_recall: 0.7515
Epoch 30/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4197 - recall: 0.7955 - val_loss: 0.4999 - val_recall: 0.7577
Epoch 31/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4108 - recall: 0.8076 - val_loss: 0.5041 - val_recall: 0.7669
Epoch 32/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4203 - recall: 0.7968 - val_loss: 0.4935 - val_recall: 0.7515
Epoch 33/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4160 - recall: 0.8049 - val_loss: 0.4926 - val_recall: 0.7362
In [72]:
plot(history,'loss')
No description has been provided for this image
In [73]:
plot(history,'recall')
No description has been provided for this image
In [74]:
val_pred = model_7.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
40/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
              precision    recall  f1-score   support

         0.0       0.92      0.77      0.84      1274
         1.0       0.45      0.74      0.56       326

    accuracy                           0.76      1600
   macro avg       0.69      0.75      0.70      1600
weighted avg       0.82      0.76      0.78      1600

No description has been provided for this image
In [77]:
results.loc[7] = [2, "64,32", "tanh,tanh", 50, 32, "SGD", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [78]:
results
Out[78]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04
6 2 128,64 relu,relu 50 8 Adam 0, - xavier L2,L1 0.550212 0.607145 0.777473 0.779141 681.80
7 2 64,32 tanh,tanh 50 32 SGD 0.001, - xavier - 0.417627 0.492607 0.799843 0.736196 404.54
In [79]:
tf.keras.backend.clear_session()

model_8 = Sequential()
model_8.add(Dense(32, activation="sigmoid", input_dim=X_train_smote.shape[1])) #sigmoid
model_8.add(Dense(16, activation="sigmoid")) #sigmoid
model_8.add(Dense(1, activation='sigmoid'))  #sigmoid

optimizer = tf.keras.optimizers.SGD(learning_rate=0.0001, momentum=0.9) #low learning rate
model_8.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

early_stopping = EarlyStopping(monitor='val_recall', patience=5, restore_best_weights=True)

start = time.time()
history = model_8.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    batch_size=32, epochs=50,
                    callbacks=[early_stopping])
end = time.time()
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.7095 - recall: 0.0000e+00
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.7095 - recall: 0.0000e+00 - val_loss: 0.6368 - val_recall: 0.0000e+00
Epoch 2/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6992 - recall: 0.0028 - val_loss: 0.6624 - val_recall: 0.0245
Epoch 3/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6955 - recall: 0.0374 - val_loss: 0.6772 - val_recall: 0.1319
Epoch 4/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6941 - recall: 0.1626 - val_loss: 0.6862 - val_recall: 0.2577
Epoch 5/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6933 - recall: 0.3287 - val_loss: 0.6916 - val_recall: 0.3681
Epoch 6/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6932 - recall: 0.4331 - val_loss: 0.6949 - val_recall: 0.4724
Epoch 7/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6924 - recall: 0.5261 - val_loss: 0.6969 - val_recall: 0.5368
Epoch 8/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6917 - recall: 0.5781 - val_loss: 0.6977 - val_recall: 0.5798
Epoch 9/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6918 - recall: 0.6352 - val_loss: 0.6979 - val_recall: 0.5951
Epoch 10/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6919 - recall: 0.5989 - val_loss: 0.6984 - val_recall: 0.6074
Epoch 11/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6911 - recall: 0.6401 - val_loss: 0.6978 - val_recall: 0.6104
Epoch 12/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6910 - recall: 0.6598 - val_loss: 0.6972 - val_recall: 0.6074
Epoch 13/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6899 - recall: 0.6381 - val_loss: 0.6972 - val_recall: 0.6196
Epoch 14/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6899 - recall: 0.6526 - val_loss: 0.6970 - val_recall: 0.6258
Epoch 15/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6901 - recall: 0.6726 - val_loss: 0.6966 - val_recall: 0.6288
Epoch 16/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6893 - recall: 0.6541 - val_loss: 0.6967 - val_recall: 0.6442
Epoch 17/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6887 - recall: 0.6772 - val_loss: 0.6952 - val_recall: 0.6288
Epoch 18/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6888 - recall: 0.6718 - val_loss: 0.6946 - val_recall: 0.6258
Epoch 19/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6882 - recall: 0.6307 - val_loss: 0.6958 - val_recall: 0.6687
Epoch 20/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6876 - recall: 0.6997 - val_loss: 0.6954 - val_recall: 0.6718
Epoch 21/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6876 - recall: 0.6828 - val_loss: 0.6951 - val_recall: 0.6810
Epoch 22/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6872 - recall: 0.6730 - val_loss: 0.6945 - val_recall: 0.6748
Epoch 23/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6865 - recall: 0.6970 - val_loss: 0.6942 - val_recall: 0.6840
Epoch 24/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6867 - recall: 0.6705 - val_loss: 0.6934 - val_recall: 0.6687
Epoch 25/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6861 - recall: 0.6740 - val_loss: 0.6936 - val_recall: 0.6963
Epoch 26/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6859 - recall: 0.6716 - val_loss: 0.6931 - val_recall: 0.6902
Epoch 27/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6852 - recall: 0.6955 - val_loss: 0.6922 - val_recall: 0.6748
Epoch 28/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6847 - recall: 0.6997 - val_loss: 0.6923 - val_recall: 0.6933
Epoch 29/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6849 - recall: 0.6770 - val_loss: 0.6921 - val_recall: 0.6963
Epoch 30/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6841 - recall: 0.6901 - val_loss: 0.6921 - val_recall: 0.7117
Epoch 31/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6837 - recall: 0.7015 - val_loss: 0.6917 - val_recall: 0.7117
Epoch 32/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6828 - recall: 0.6971 - val_loss: 0.6915 - val_recall: 0.7178
Epoch 33/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6826 - recall: 0.7202 - val_loss: 0.6910 - val_recall: 0.7055
Epoch 34/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6822 - recall: 0.6846 - val_loss: 0.6915 - val_recall: 0.7423
Epoch 35/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6823 - recall: 0.7073 - val_loss: 0.6916 - val_recall: 0.7515
Epoch 36/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6819 - recall: 0.7012 - val_loss: 0.6912 - val_recall: 0.7423
Epoch 37/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6813 - recall: 0.7241 - val_loss: 0.6905 - val_recall: 0.7270
Epoch 38/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 13s 39ms/step - loss: 0.6809 - recall: 0.7082 - val_loss: 0.6900 - val_recall: 0.7239
Epoch 39/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6809 - recall: 0.7052 - val_loss: 0.6894 - val_recall: 0.7178
Epoch 40/50
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6805 - recall: 0.6964 - val_loss: 0.6893 - val_recall: 0.7239
In [80]:
plot(history,'loss')
No description has been provided for this image
In [81]:
plot(history,'recall')
No description has been provided for this image
In [82]:
val_pred = model_8.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
39/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
              precision    recall  f1-score   support

         0.0       0.88      0.46      0.60      1274
         1.0       0.26      0.75      0.39       326

    accuracy                           0.52      1600
   macro avg       0.57      0.60      0.49      1600
weighted avg       0.75      0.52      0.56      1600

No description has been provided for this image
In [83]:
results.loc[8] = [2, "32,16", "sigmoid,sigmoid", 50, 32, "SGD", "0.0001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [84]:
results
Out[84]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04
6 2 128,64 relu,relu 50 8 Adam 0, - xavier L2,L1 0.550212 0.607145 0.777473 0.779141 681.80
7 2 64,32 tanh,tanh 50 32 SGD 0.001, - xavier - 0.417627 0.492607 0.799843 0.736196 404.54
8 2 32,16 sigmoid,sigmoid 50 32 SGD 0.0001, - xavier - 0.680181 0.689308 0.696625 0.723926 490.79
In [85]:
tf.keras.backend.clear_session()

model_9 = Sequential()
model_9.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1]))
model_9.add(Dense(64, activation="relu"))
model_9.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_9.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

start = time.time()
history = model_9.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    batch_size=32, epochs=100,
                    callbacks=[early_stopping]) # Pass the callback to fit
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.6697 - recall: 0.8128
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6696 - recall: 0.8126 - val_loss: 0.6018 - val_recall: 0.6288
Epoch 2/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6014 - recall: 0.6672 - val_loss: 0.5683 - val_recall: 0.6963
Epoch 3/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.5607 - recall: 0.6919 - val_loss: 0.5514 - val_recall: 0.7239
Epoch 4/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.5383 - recall: 0.7157 - val_loss: 0.5463 - val_recall: 0.7546
Epoch 5/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.5244 - recall: 0.7325 - val_loss: 0.5214 - val_recall: 0.7423
Epoch 6/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5013 - recall: 0.7385 - val_loss: 0.5166 - val_recall: 0.7638
Epoch 7/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4806 - recall: 0.7484 - val_loss: 0.5325 - val_recall: 0.7853
Epoch 8/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4718 - recall: 0.7604 - val_loss: 0.5369 - val_recall: 0.7945
Epoch 9/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4568 - recall: 0.7671 - val_loss: 0.5494 - val_recall: 0.8098
Epoch 10/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4447 - recall: 0.7776 - val_loss: 0.5659 - val_recall: 0.8190
Epoch 11/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4364 - recall: 0.7866 - val_loss: 0.5565 - val_recall: 0.8067
Time taken in seconds  135.95033502578735
In [86]:
plot(history,'loss')
No description has been provided for this image
In [87]:
plot(history,'recall')
No description has been provided for this image
In [88]:
val_pred = model_9.predict(X_val)
val_pred = (val_pred > 0.5)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
39/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
              precision    recall  f1-score   support

         0.0       0.92      0.74      0.82      1274
         1.0       0.43      0.76      0.55       326

    accuracy                           0.75      1600
   macro avg       0.68      0.75      0.69      1600
weighted avg       0.82      0.75      0.77      1600

No description has been provided for this image

this is fairly balanced i kind of like it its not prefect but its no overly biased towards recall

In [89]:
results.loc[9] = [2, "128,64", "relu,relu", 150, 32, "SGD", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [90]:
results
Out[90]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04
6 2 128,64 relu,relu 50 8 Adam 0, - xavier L2,L1 0.550212 0.607145 0.777473 0.779141 681.80
7 2 64,32 tanh,tanh 50 32 SGD 0.001, - xavier - 0.417627 0.492607 0.799843 0.736196 404.54
8 2 32,16 sigmoid,sigmoid 50 32 SGD 0.0001, - xavier - 0.680181 0.689308 0.696625 0.723926 490.79
9 2 128,64 relu,relu 150 32 SGD 0.001, - xavier - 0.434032 0.556475 0.788265 0.806748 135.95

we need to give the model more epochs and more momentum and a small batch size

In [94]:
tf.keras.backend.clear_session()

model_10 = Sequential()
model_10.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1]))
model_10.add(Dense(64, activation="relu"))
model_10.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.999) #larger momentum
model_10.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])


start = time.time()
history = model_10.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    batch_size=16, epochs=100, #smaller batch size
                    )
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.6045 - recall: 0.6807
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6044 - recall: 0.6807 - val_loss: 0.7312 - val_recall: 0.8374
Epoch 2/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4725 - recall: 0.7751 - val_loss: 0.5062 - val_recall: 0.7055
Epoch 3/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4443 - recall: 0.7705 - val_loss: 0.8109 - val_recall: 0.8742
Epoch 4/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4327 - recall: 0.8315 - val_loss: 0.6591 - val_recall: 0.8282
Epoch 5/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4140 - recall: 0.7881 - val_loss: 0.6423 - val_recall: 0.8681
Epoch 6/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4434 - recall: 0.8479 - val_loss: 0.6399 - val_recall: 0.8497
Epoch 7/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4276 - recall: 0.8049 - val_loss: 0.6787 - val_recall: 0.7822
Epoch 8/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4205 - recall: 0.7256 - val_loss: 0.7436 - val_recall: 0.9080
Epoch 9/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4239 - recall: 0.8807 - val_loss: 0.9416 - val_recall: 0.7699
Epoch 10/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4299 - recall: 0.6948 - val_loss: 0.6749 - val_recall: 0.7117
Epoch 11/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4057 - recall: 0.7039 - val_loss: 0.6068 - val_recall: 0.7975
Epoch 12/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4152 - recall: 0.7237 - val_loss: 0.7627 - val_recall: 0.8252
Epoch 13/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.3938 - recall: 0.7537 - val_loss: 1.0532 - val_recall: 0.8282
Epoch 14/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4114 - recall: 0.7439 - val_loss: 0.9321 - val_recall: 0.9325
Epoch 15/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4185 - recall: 0.9097 - val_loss: 0.6232 - val_recall: 0.7914
Epoch 16/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4302 - recall: 0.7005 - val_loss: 0.7076 - val_recall: 0.9233
Epoch 17/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4428 - recall: 0.9022 - val_loss: 1.0876 - val_recall: 0.9387
Epoch 18/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4560 - recall: 0.9216 - val_loss: 1.1865 - val_recall: 0.9724
Epoch 19/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4752 - recall: 0.8886 - val_loss: 0.8114 - val_recall: 0.9724
Epoch 20/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.5016 - recall: 0.9049 - val_loss: 0.8612 - val_recall: 0.9632
Epoch 21/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4948 - recall: 0.8929 - val_loss: 0.8160 - val_recall: 0.9264
Epoch 22/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4903 - recall: 0.8770 - val_loss: 0.5357 - val_recall: 0.1288
Epoch 23/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.5353 - recall: 0.7547 - val_loss: 0.7762 - val_recall: 0.9479
Epoch 24/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6011 - recall: 0.8326 - val_loss: 0.7171 - val_recall: 0.9693
Epoch 25/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.5854 - recall: 0.8631 - val_loss: 1.0178 - val_recall: 0.8313
Epoch 26/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.5475 - recall: 0.6876 - val_loss: 1.2611 - val_recall: 0.7638
Epoch 27/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.5512 - recall: 0.6260 - val_loss: 1.3986 - val_recall: 0.7577
Epoch 28/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6342 - recall: 0.5654 - val_loss: 1.5879 - val_recall: 0.4724
Epoch 29/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6160 - recall: 0.3916 - val_loss: 0.5662 - val_recall: 0.1043
Epoch 30/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6361 - recall: 0.5912 - val_loss: 0.8483 - val_recall: 0.9877
Epoch 31/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6178 - recall: 0.4242 - val_loss: 0.7379 - val_recall: 0.9540
Epoch 32/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6735 - recall: 0.9506 - val_loss: 0.6412 - val_recall: 0.9847
Epoch 33/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6642 - recall: 0.8124 - val_loss: 0.6228 - val_recall: 0.9417
Epoch 34/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6676 - recall: 0.6253 - val_loss: 0.6481 - val_recall: 0.9356
Epoch 35/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6406 - recall: 0.9130 - val_loss: 0.6469 - val_recall: 0.9663
Epoch 36/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6392 - recall: 0.6713 - val_loss: 0.6946 - val_recall: 0.9755
Epoch 37/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6433 - recall: 0.9738 - val_loss: 0.7107 - val_recall: 0.9632
Epoch 38/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6327 - recall: 0.9615 - val_loss: 0.6083 - val_recall: 0.0092
Epoch 39/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 38ms/step - loss: 0.6663 - recall: 0.7649 - val_loss: 0.6283 - val_recall: 0.9939
Epoch 40/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6404 - recall: 0.9860 - val_loss: 0.6256 - val_recall: 0.9847
Epoch 41/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6334 - recall: 0.9725 - val_loss: 0.6813 - val_recall: 0.9755
Epoch 42/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6538 - recall: 0.9128 - val_loss: 0.7284 - val_recall: 0.9969
Epoch 43/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6911 - recall: 0.6940 - val_loss: 0.6519 - val_recall: 0.0031
Epoch 44/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6818 - recall: 0.5046 - val_loss: 0.7775 - val_recall: 0.9939
Epoch 45/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6699 - recall: 0.6966 - val_loss: 0.6893 - val_recall: 0.9847
Epoch 46/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6728 - recall: 0.9932 - val_loss: 0.6783 - val_recall: 0.9755
Epoch 47/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6515 - recall: 0.9215 - val_loss: 0.7836 - val_recall: 0.9785
Epoch 48/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6500 - recall: 0.8488 - val_loss: 0.6215 - val_recall: 0.9417
Epoch 49/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6213 - recall: 0.9765 - val_loss: 0.9208 - val_recall: 0.9908
Epoch 50/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6576 - recall: 0.7119 - val_loss: 0.6144 - val_recall: 0.0000e+00
Epoch 51/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6897 - recall: 0.3887 - val_loss: 0.8826 - val_recall: 0.9939
Epoch 52/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6868 - recall: 0.6323 - val_loss: 0.6340 - val_recall: 0.0000e+00
Epoch 53/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6606 - recall: 0.6389 - val_loss: 0.7373 - val_recall: 0.9785
Epoch 54/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6393 - recall: 0.9858 - val_loss: 0.6918 - val_recall: 0.9663
Epoch 55/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6472 - recall: 0.9193 - val_loss: 0.7436 - val_recall: 0.9816
Epoch 56/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6243 - recall: 0.8104 - val_loss: 0.5900 - val_recall: 0.9417
Epoch 57/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6413 - recall: 0.9114 - val_loss: 0.7386 - val_recall: 0.9785
Epoch 58/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6537 - recall: 0.8111 - val_loss: 0.6396 - val_recall: 0.9785
Epoch 59/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6557 - recall: 0.3432 - val_loss: 0.6838 - val_recall: 0.9816
Epoch 60/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6380 - recall: 0.8517 - val_loss: 0.8151 - val_recall: 0.9847
Epoch 61/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6135 - recall: 0.4520 - val_loss: 0.7738 - val_recall: 0.9755
Epoch 62/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6142 - recall: 0.8386 - val_loss: 0.7250 - val_recall: 0.9755
Epoch 63/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6311 - recall: 0.5310 - val_loss: 0.8438 - val_recall: 0.9663
Epoch 64/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6125 - recall: 0.9784 - val_loss: 0.8696 - val_recall: 0.9540
Epoch 65/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6045 - recall: 0.9431 - val_loss: 0.9773 - val_recall: 0.9356
Epoch 66/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6360 - recall: 0.9299 - val_loss: 1.1951 - val_recall: 0.9724
Epoch 67/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6992 - recall: 0.9979 - val_loss: 1.0997 - val_recall: 0.9755
Epoch 68/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6199 - recall: 0.9883 - val_loss: 0.9158 - val_recall: 0.9663
Epoch 69/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6254 - recall: 0.9857 - val_loss: 0.7324 - val_recall: 0.9693
Epoch 70/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6085 - recall: 0.9934 - val_loss: 0.8071 - val_recall: 0.9724
Epoch 71/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6170 - recall: 0.9935 - val_loss: 0.8005 - val_recall: 0.9724
Epoch 72/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6212 - recall: 0.9932 - val_loss: 0.7184 - val_recall: 0.9663
Epoch 73/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 38ms/step - loss: 0.6190 - recall: 0.6870 - val_loss: 0.8518 - val_recall: 0.9693
Epoch 74/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6153 - recall: 0.9926 - val_loss: 0.8498 - val_recall: 0.0000e+00
Epoch 75/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6787 - recall: 0.7039 - val_loss: 1.0993 - val_recall: 0.9755
Epoch 76/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6475 - recall: 0.6497 - val_loss: 0.7246 - val_recall: 0.0000e+00
Epoch 77/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6609 - recall: 0.5842 - val_loss: 0.9813 - val_recall: 0.9847
Epoch 78/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6537 - recall: 0.7260 - val_loss: 0.6909 - val_recall: 0.0000e+00
Epoch 79/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6576 - recall: 0.6112 - val_loss: 0.7906 - val_recall: 0.9877
Epoch 80/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6542 - recall: 0.8324 - val_loss: 0.6328 - val_recall: 0.0000e+00
Epoch 81/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6538 - recall: 0.6453 - val_loss: 0.7568 - val_recall: 0.9877
Epoch 82/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6277 - recall: 0.9968 - val_loss: 0.6605 - val_recall: 0.9663
Epoch 83/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6223 - recall: 0.9933 - val_loss: 0.7700 - val_recall: 0.9785
Epoch 84/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6191 - recall: 0.9967 - val_loss: 0.7342 - val_recall: 0.9785
Epoch 85/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6125 - recall: 0.9795 - val_loss: 0.7667 - val_recall: 0.9816
Epoch 86/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6243 - recall: 0.9983 - val_loss: 0.6680 - val_recall: 0.9847
Epoch 87/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6329 - recall: 1.0000 - val_loss: 0.7957 - val_recall: 0.9816
Epoch 88/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6316 - recall: 0.9667 - val_loss: 1.0267 - val_recall: 0.9816
Epoch 89/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6393 - recall: 0.8865 - val_loss: 0.7234 - val_recall: 0.9785
Epoch 90/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6481 - recall: 0.5827 - val_loss: 0.8466 - val_recall: 0.9847
Epoch 91/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6564 - recall: 0.7933 - val_loss: 0.7983 - val_recall: 0.0000e+00
Epoch 92/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6536 - recall: 0.4721 - val_loss: 1.0292 - val_recall: 0.9847
Epoch 93/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6260 - recall: 0.8216 - val_loss: 0.6948 - val_recall: 0.9755
Epoch 94/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6094 - recall: 0.9974 - val_loss: 0.9607 - val_recall: 0.9755
Epoch 95/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6315 - recall: 0.9969 - val_loss: 0.7830 - val_recall: 0.9816
Epoch 96/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6338 - recall: 0.9939 - val_loss: 0.7495 - val_recall: 0.0000e+00
Epoch 97/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6284 - recall: 0.6500 - val_loss: 0.8744 - val_recall: 0.9724
Epoch 98/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6140 - recall: 0.9968 - val_loss: 0.8675 - val_recall: 0.9693
Epoch 99/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6006 - recall: 0.9937 - val_loss: 0.8863 - val_recall: 0.9571
Epoch 100/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6087 - recall: 0.9953 - val_loss: 0.8465 - val_recall: 0.9571
Time taken in seconds  2449.964891910553

that was too many epochs we need more epoch but not that many.... maybe early stopping of 15 will do it or a max of 50 epochs

In [95]:
plot(history,'loss')
No description has been provided for this image
In [96]:
plot(history,'recall')
No description has been provided for this image

this is very interesting but i have no idea how interpret this....

In [97]:
val_pred = model_10.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
37/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
              precision    recall  f1-score   support

         0.0       0.97      0.32      0.49      1274
         1.0       0.27      0.96      0.42       326

    accuracy                           0.45      1600
   macro avg       0.62      0.64      0.45      1600
weighted avg       0.82      0.45      0.47      1600

No description has been provided for this image
In [98]:
results.loc[10] = [2, "128,64", "relu,relu", 100, 16, "SGD", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [99]:
results
Out[99]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04
6 2 128,64 relu,relu 50 8 Adam 0, - xavier L2,L1 0.550212 0.607145 0.777473 0.779141 681.80
7 2 64,32 tanh,tanh 50 32 SGD 0.001, - xavier - 0.417627 0.492607 0.799843 0.736196 404.54
8 2 32,16 sigmoid,sigmoid 50 32 SGD 0.0001, - xavier - 0.680181 0.689308 0.696625 0.723926 490.79
9 2 128,64 relu,relu 150 32 SGD 0.001, - xavier - 0.434032 0.556475 0.788265 0.806748 135.95
10 2 128,64 relu,relu 100 16 SGD 0.001, - xavier - 0.598579 0.846530 0.994309 0.957055 2449.96

our precision f1 scores are bad our recall is good but we need to see if we can get a model that would create better resource allocation for when it comes to our strategies to deal with churning

In [100]:
tf.keras.backend.clear_session()

model_11 = Sequential()
# Input layer with L2 regularization and Glorot uniform initialization
model_11.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1],
                 kernel_initializer='glorot_uniform',  # Xavier Glorot
                 kernel_regularizer=regularizers.l2(0.001)))
model_11.add(BatchNormalization())
model_11.add(Dropout(0.5)) #dropout
model_11.add(Dense(64, activation="relu", kernel_initializer='glorot_uniform')) # Xavier Glorot
model_11.add(BatchNormalization())
model_11.add(Dropout(0.5)) #dropout
model_11.add(Dense(1, activation='sigmoid', kernel_initializer='glorot_uniform')) # Xavier Glorot

# Calculate class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.99)
model_11.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

#EarlyStopping patience=15 but on loss function
early_stopping = EarlyStopping(monitor='val_loss', patience=15, restore_best_weights=True)

start = time.time()
history = model_11.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=16, epochs=50,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 68ms/step - loss: 0.9731 - recall: 0.8383
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.9729 - recall: 0.8384 - val_loss: 1.2114 - val_recall: 0.9755
Epoch 2/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6950 - recall: 0.9488 - val_loss: 1.1756 - val_recall: 0.9632
Epoch 3/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6452 - recall: 0.9465 - val_loss: 1.1012 - val_recall: 0.9540
Epoch 4/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6397 - recall: 0.9455 - val_loss: 1.0429 - val_recall: 0.9724
Epoch 5/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6272 - recall: 0.9449 - val_loss: 0.9858 - val_recall: 0.9755
Epoch 6/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6187 - recall: 0.9460 - val_loss: 1.1074 - val_recall: 0.9663
Epoch 7/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 47s 73ms/step - loss: 0.6091 - recall: 0.9431 - val_loss: 1.0829 - val_recall: 0.9724
Epoch 8/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5900 - recall: 0.9473 - val_loss: 1.0255 - val_recall: 0.9571
Epoch 9/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5932 - recall: 0.9496 - val_loss: 1.1489 - val_recall: 0.9755
Epoch 10/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5877 - recall: 0.9429 - val_loss: 1.1258 - val_recall: 0.9724
Epoch 11/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5837 - recall: 0.9487 - val_loss: 1.0471 - val_recall: 0.9601
Epoch 12/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 73ms/step - loss: 0.5732 - recall: 0.9485 - val_loss: 1.1743 - val_recall: 0.9785
Epoch 13/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 47s 73ms/step - loss: 0.5703 - recall: 0.9538 - val_loss: 1.0951 - val_recall: 0.9785
Epoch 14/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5864 - recall: 0.9617 - val_loss: 1.1145 - val_recall: 0.9785
Epoch 15/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5777 - recall: 0.9554 - val_loss: 1.0810 - val_recall: 0.9693
Epoch 16/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5665 - recall: 0.9500 - val_loss: 1.0420 - val_recall: 0.9509
Epoch 17/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5720 - recall: 0.9494 - val_loss: 1.0614 - val_recall: 0.9663
Epoch 18/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5784 - recall: 0.9490 - val_loss: 1.0778 - val_recall: 0.9693
Epoch 19/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5736 - recall: 0.9514 - val_loss: 1.1204 - val_recall: 0.9724
Epoch 20/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5814 - recall: 0.9475 - val_loss: 1.2318 - val_recall: 0.9693
Time taken in seconds  922.238805770874

with glorot we may be able to speed things up by bringing back early stopping with patience 10 or even 5!

In [101]:
plot(history,'loss')
No description has been provided for this image
In [102]:
plot(history,'recall')
No description has been provided for this image
In [103]:
val_pred = model_11.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
22/50 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
              precision    recall  f1-score   support

         0.0       0.98      0.32      0.48      1274
         1.0       0.27      0.98      0.42       326

    accuracy                           0.45      1600
   macro avg       0.62      0.65      0.45      1600
weighted avg       0.84      0.45      0.47      1600

No description has been provided for this image

so far in our best model we are still predicting a lot of people will churn that will not that we will be wasting resources trying to stop from churning despite all these fancy hyperparameters the precision and accuracy are low

In [104]:
results.loc[11] = [2, "128,64", "relu,relu", 50, 16, "SGD", "0.001, -", "xavier", "L2,N/A", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [105]:
results
Out[105]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04
6 2 128,64 relu,relu 50 8 Adam 0, - xavier L2,L1 0.550212 0.607145 0.777473 0.779141 681.80
7 2 64,32 tanh,tanh 50 32 SGD 0.001, - xavier - 0.417627 0.492607 0.799843 0.736196 404.54
8 2 32,16 sigmoid,sigmoid 50 32 SGD 0.0001, - xavier - 0.680181 0.689308 0.696625 0.723926 490.79
9 2 128,64 relu,relu 150 32 SGD 0.001, - xavier - 0.434032 0.556475 0.788265 0.806748 135.95
10 2 128,64 relu,relu 100 16 SGD 0.001, - xavier - 0.598579 0.846530 0.994309 0.957055 2449.96
11 2 128,64 relu,relu 50 16 SGD 0.001, - xavier L2,N/A 0.575172 1.231755 0.947214 0.969325 922.24

this is still better but the precision and f1 remain very low.

im going to lower the kernal regulizer parameter so that the model can hopefully capture more complexity.

also, relu can sometimes lead to "dead neurons" which can hinder learning. it would be a good idea to try LeakyReLU or ELU.

we could increase model complexity CAREFULLY

OR HIGHER CLASS WEIGHTS

lastly it may be a good idea to try He initilaizer

In [ ]:
from tensorflow.keras.layers import LeakyReLU
In [106]:
tf.keras.backend.clear_session()

model_12 = Sequential()

model_12.add(Dense(128, activation=LeakyReLU(alpha=0.1),
                 input_dim=X_train_smote.shape[1],
                 kernel_initializer='glorot_uniform',  # Xavier Glorot
                 kernel_regularizer=regularizers.l2(0.0001)))  # L2 regularization
model_12.add(BatchNormalization())  # Batch Normalization
model_12.add(Dropout(0.5))  # Dropout

# Hidden layer
model_12.add(Dense(64, activation=LeakyReLU(alpha=0.1),
                 kernel_initializer='glorot_uniform',  # Xavier Glorot
                 kernel_regularizer=regularizers.l2(0.0001)))  # L2 regularization
model_12.add(BatchNormalization())  # Batch Normalization
model_12.add(Dropout(0.5))  # Dropout

model_12.add(Dense(32, activation='relu', kernel_initializer='glorot_uniform'))

model_12.add(Dense(1, activation='sigmoid', kernel_initializer='glorot_uniform'))

class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_12.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

# Add EarlyStopping callback with patience=5
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

start = time.time()
history = model_12.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=32, epochs=100,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 75ms/step - loss: 0.9232 - recall: 0.8638
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.9228 - recall: 0.8640 - val_loss: 0.9511 - val_recall: 0.9877
Epoch 2/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.7133 - recall: 0.9685 - val_loss: 0.9980 - val_recall: 0.9785
Epoch 3/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.6824 - recall: 0.9675 - val_loss: 0.9633 - val_recall: 0.9663
Epoch 4/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.6730 - recall: 0.9663 - val_loss: 0.9821 - val_recall: 0.9693
Epoch 5/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.6571 - recall: 0.9668 - val_loss: 0.9764 - val_recall: 0.9632
Epoch 6/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.6575 - recall: 0.9629 - val_loss: 0.9730 - val_recall: 0.9601
Time taken in seconds  151.72719502449036
In [107]:
plot(history,'loss')
No description has been provided for this image
In [108]:
plot(history,'recall')
No description has been provided for this image
In [109]:
val_pred = model_12.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
13/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
              precision    recall  f1-score   support

         0.0       0.96      0.07      0.13      1274
         1.0       0.21      0.99      0.35       326

    accuracy                           0.26      1600
   macro avg       0.59      0.53      0.24      1600
weighted avg       0.81      0.26      0.18      1600

No description has been provided for this image

recall was not impacted but our preeciion on target variable and the accuracy are worse

In [110]:
results.loc[12] = [3, "128,64,32", "leakyrelu,leakyrelu,relu", 100, 32, "SGD", "0.001, -", "xavier", "L2,L2,N/A", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [111]:
results
Out[111]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04
6 2 128,64 relu,relu 50 8 Adam 0, - xavier L2,L1 0.550212 0.607145 0.777473 0.779141 681.80
7 2 64,32 tanh,tanh 50 32 SGD 0.001, - xavier - 0.417627 0.492607 0.799843 0.736196 404.54
8 2 32,16 sigmoid,sigmoid 50 32 SGD 0.0001, - xavier - 0.680181 0.689308 0.696625 0.723926 490.79
9 2 128,64 relu,relu 150 32 SGD 0.001, - xavier - 0.434032 0.556475 0.788265 0.806748 135.95
10 2 128,64 relu,relu 100 16 SGD 0.001, - xavier - 0.598579 0.846530 0.994309 0.957055 2449.96
11 2 128,64 relu,relu 50 16 SGD 0.001, - xavier L2,N/A 0.575172 1.231755 0.947214 0.969325 922.24
12 3 128,64,32 leakyrelu,leakyrelu,relu 100 32 SGD 0.001, - xavier L2,L2,N/A 0.643929 0.973020 0.965463 0.960123 151.73

add more agressive class weights

In [113]:
tf.keras.backend.clear_session()

model_13 = Sequential()

model_13.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train_smote.shape[1],
                 kernel_regularizer=regularizers.l2(0.0001)))
model_13.add(Dropout(0.2))
model_13.add(Dense(64, activation=LeakyReLU(alpha=0.1)))
model_13.add(Dropout(0.2))
model_13.add(Dense(32, activation='relu'))
model_13.add(Dense(1, activation='sigmoid'))

class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]} #bit more aggressive

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)

model_13.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])

early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

start = time.time()
history = model_13.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=16, epochs=100,  #smaller batch sizes
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 53ms/step - loss: 0.8289 - recall: 0.9091
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

637/637 ━━━━━━━━━━━━━━━━━━━━ 37s 57ms/step - loss: 0.8287 - recall: 0.9092 - val_loss: 1.0147 - val_recall: 1.0000
Epoch 2/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 37s 58ms/step - loss: 0.6650 - recall: 0.9884 - val_loss: 0.9578 - val_recall: 0.9540
Epoch 3/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.6284 - recall: 0.9711 - val_loss: 0.9190 - val_recall: 0.9417
Epoch 4/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.6048 - recall: 0.9571 - val_loss: 0.9286 - val_recall: 0.9509
Epoch 5/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.5780 - recall: 0.9577 - val_loss: 0.9301 - val_recall: 0.9540
Epoch 6/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.5650 - recall: 0.9557 - val_loss: 0.9820 - val_recall: 0.9632
Epoch 7/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 37s 58ms/step - loss: 0.5467 - recall: 0.9501 - val_loss: 1.0206 - val_recall: 0.9632
Epoch 8/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.5400 - recall: 0.9520 - val_loss: 0.9838 - val_recall: 0.9632
Time taken in seconds  291.93304467201233
In [114]:
plot(history,'loss')
No description has been provided for this image
In [115]:
plot(history,'recall')
No description has been provided for this image
In [116]:
val_pred = model_13.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
28/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
              precision    recall  f1-score   support

         0.0       0.95      0.31      0.47      1274
         1.0       0.26      0.94      0.41       326

    accuracy                           0.44      1600
   macro avg       0.61      0.63      0.44      1600
weighted avg       0.81      0.44      0.45      1600

No description has been provided for this image
In [117]:
results.loc[13] = [3, "128,64,32", "leakyrelu,leakyrelu,relu", 100, 16, "SGD", "0.001, -", "xavier", "L2,N/A", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
In [118]:
results
Out[118]:
# hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size optimizer learning rate initializer regularizer train loss validation loss train recall validation recall time (secs)
0 2 128,64 relu,relu 10 16 sgd 0.001, - xavier - 0.459178 0.455843 0.000000 0.003067 143.87
1 2 64,32 relu,relu 50 16 Adam 0, - xavier - 0.261802 0.385848 0.608129 0.457055 606.90
3 2 128,64 relu,relu 100 32 SGD 0.001, - xavier - 0.389204 0.717597 0.816130 0.858896 332.36
4 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.574763 0.659100 0.776295 0.797546 264.18
5 2 128,64 relu,relu 100 32 Adam 0, - xavier L2,L1 0.640411 0.693025 0.772174 0.806748 198.04
6 2 128,64 relu,relu 50 8 Adam 0, - xavier L2,L1 0.550212 0.607145 0.777473 0.779141 681.80
7 2 64,32 tanh,tanh 50 32 SGD 0.001, - xavier - 0.417627 0.492607 0.799843 0.736196 404.54
8 2 32,16 sigmoid,sigmoid 50 32 SGD 0.0001, - xavier - 0.680181 0.689308 0.696625 0.723926 490.79
9 2 128,64 relu,relu 150 32 SGD 0.001, - xavier - 0.434032 0.556475 0.788265 0.806748 135.95
10 2 128,64 relu,relu 100 16 SGD 0.001, - xavier - 0.598579 0.846530 0.994309 0.957055 2449.96
11 2 128,64 relu,relu 50 16 SGD 0.001, - xavier L2,N/A 0.575172 1.231755 0.947214 0.969325 922.24
12 3 128,64,32 leakyrelu,leakyrelu,relu 100 32 SGD 0.001, - xavier L2,L2,N/A 0.643929 0.973020 0.965463 0.960123 151.73
13 3 128,64,32 leakyrelu,leakyrelu,relu 100 16 SGD 0.001, - xavier L2,N/A 0.537729 0.983835 0.952512 0.963190 291.93

more agressive class weight

In [125]:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Dense, Dropout, LeakyReLU, Input
from tensorflow.keras.models import Sequential
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.utils.class_weight import compute_class_weight
from tensorflow.keras import regularizers
import time
import numpy as np

tf.keras.backend.clear_session()

# Custom F1 score metric for the minority class (target variable)
def f1_minority(y_true, y_pred):
    y_true = tf.cast(y_true, tf.float32)
    y_pred = tf.cast(y_pred, tf.float32)

    tp = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    pp = K.sum(K.round(K.clip(y_pred, 0, 1)))
    fp = pp - tp
    fn = K.sum(K.round(K.clip(y_true, 0, 1))) - tp

    precision = tp / (tp + fp + K.epsilon())
    recall = tp / (tp + fn + K.epsilon())

    f1_val = 2 * ((precision * recall) / (precision + recall + K.epsilon()))
    return f1_val
In [126]:
model_14 = Sequential()
#
model_14.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train_smote.shape[1],
                 kernel_initializer='glorot_uniform',
                 kernel_regularizer=regularizers.l2(0.0001)))
model_14.add(Dropout(0.2))

model_14.add(Dense(64, activation=LeakyReLU(alpha=0.1),
                 kernel_initializer='glorot_uniform',
                 kernel_regularizer=regularizers.l2(0.0001)))
model_14.add(Dropout(0.2))

model_14.add(Dense(32, activation='relu', kernel_initializer='glorot_uniform'))
model_14.add(Dense(1, activation='sigmoid', kernel_initializer='glorot_uniform'))

class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: 1, 1: 1.5} #slightly more aggressive

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)

# Compile the model with the custom F1 score metric
model_14.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[f1_minority])

early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

start = time.time()
history = model_14.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=16, epochs=100,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
636/637 ━━━━━━━━━━━━━━━━━━━━ 0s 49ms/step - f1_minority: 0.6327 - loss: 0.8170
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 52ms/step - f1_minority: 0.6329 - loss: 0.8169 - val_f1_minority: 0.4314 - val_loss: 0.7209
Epoch 2/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 51ms/step - f1_minority: 0.7186 - loss: 0.7075 - val_f1_minority: 0.4467 - val_loss: 0.6953
Epoch 3/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 52ms/step - f1_minority: 0.7386 - loss: 0.6634 - val_f1_minority: 0.4564 - val_loss: 0.6698
Epoch 4/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 32s 51ms/step - f1_minority: 0.7466 - loss: 0.6492 - val_f1_minority: 0.4524 - val_loss: 0.6592
Epoch 5/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 51ms/step - f1_minority: 0.7749 - loss: 0.5898 - val_f1_minority: 0.4580 - val_loss: 0.6729
Epoch 6/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 51ms/step - f1_minority: 0.7721 - loss: 0.5809 - val_f1_minority: 0.4480 - val_loss: 0.7282
Epoch 7/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 51ms/step - f1_minority: 0.7817 - loss: 0.5772 - val_f1_minority: 0.4491 - val_loss: 0.7107
Epoch 8/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 32s 51ms/step - f1_minority: 0.7859 - loss: 0.5628 - val_f1_minority: 0.4536 - val_loss: 0.7064
Epoch 9/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 52ms/step - f1_minority: 0.7863 - loss: 0.5537 - val_f1_minority: 0.4453 - val_loss: 0.7374
Time taken in seconds  294.83499097824097
In [127]:
plot(history,'loss')
No description has been provided for this image
In [130]:
plot(history,'f1_minority')
No description has been provided for this image
In [131]:
val_pred = model_14.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
28/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
              precision    recall  f1-score   support

         0.0       0.94      0.57      0.71      1274
         1.0       0.34      0.87      0.49       326

    accuracy                           0.63      1600
   macro avg       0.64      0.72      0.60      1600
weighted avg       0.82      0.63      0.67      1600

No description has been provided for this image

this model is better at not wasting resaources butthere is opportunity cost here from worse performance on predicting churners

higher patience, higher dropout glourot a new adjusted l2 regulizer higher dropout and patience

In [133]:
model_15 = Sequential()

model_15.add(Dense(128, activation=LeakyReLU(alpha=0.1),
                    input_dim=X_train_smote.shape[1],
                    kernel_initializer='glorot_uniform',
                    kernel_regularizer=regularizers.l2(0.001)))  # L2 regularization adjusted
model_15.add(BatchNormalization())
model_15.add(Dropout(0.5))

model_15.add(Dense(64, activation=LeakyReLU(alpha=0.1),
                    kernel_initializer='glorot_uniform',
                    kernel_regularizer=regularizers.l2(0.001)))  # L2 regularization adjusted
model_15.add(BatchNormalization())
model_15.add(Dropout(0.5))

model_15.add(Dense(32, activation='relu',
                    kernel_initializer='glorot_uniform',
                    kernel_regularizer=regularizers.l2(0.001)))  # L2 regularization adjusted
model_15.add(BatchNormalization())
model_15.add(Dropout(0.5))

model_15.add(Dense(1, activation='sigmoid'))

class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: 1, 1: 1.5} #same as before

# Compile the model with a custom F1 score metric for our minority target variable class
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_15.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[f1_minority])

early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True) #decent patience

start = time.time()
history = model_15.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=16, epochs=100,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 87ms/step - f1_minority: 0.5957 - loss: 1.1143
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.5958 - loss: 1.1141 - val_f1_minority: 0.3806 - val_loss: 0.9081
Epoch 2/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 91ms/step - f1_minority: 0.6854 - loss: 0.8923 - val_f1_minority: 0.4058 - val_loss: 0.8755
Epoch 3/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.6949 - loss: 0.8710 - val_f1_minority: 0.4172 - val_loss: 0.8515
Epoch 4/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7090 - loss: 0.8507 - val_f1_minority: 0.4228 - val_loss: 0.8509
Epoch 5/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7161 - loss: 0.8366 - val_f1_minority: 0.4457 - val_loss: 0.8071
Epoch 6/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7279 - loss: 0.8226 - val_f1_minority: 0.4230 - val_loss: 0.8460
Epoch 7/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.7250 - loss: 0.8167 - val_f1_minority: 0.4283 - val_loss: 0.8336
Epoch 8/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 91ms/step - f1_minority: 0.7430 - loss: 0.7925 - val_f1_minority: 0.4372 - val_loss: 0.8042
Epoch 9/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 91ms/step - f1_minority: 0.7363 - loss: 0.7887 - val_f1_minority: 0.4381 - val_loss: 0.8132
Epoch 10/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7271 - loss: 0.7869 - val_f1_minority: 0.4350 - val_loss: 0.8053
Epoch 11/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7407 - loss: 0.7692 - val_f1_minority: 0.4550 - val_loss: 0.7628
Epoch 12/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7399 - loss: 0.7720 - val_f1_minority: 0.4346 - val_loss: 0.8022
Epoch 13/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.7510 - loss: 0.7539 - val_f1_minority: 0.4424 - val_loss: 0.7799
Epoch 14/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.7518 - loss: 0.7638 - val_f1_minority: 0.4452 - val_loss: 0.7882
Epoch 15/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7464 - loss: 0.7516 - val_f1_minority: 0.4475 - val_loss: 0.7753
Epoch 16/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.7691 - loss: 0.7205 - val_f1_minority: 0.4542 - val_loss: 0.7518
Epoch 17/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7569 - loss: 0.7293 - val_f1_minority: 0.4534 - val_loss: 0.7517
Epoch 18/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 91ms/step - f1_minority: 0.7577 - loss: 0.7110 - val_f1_minority: 0.4440 - val_loss: 0.7738
Epoch 19/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7552 - loss: 0.7164 - val_f1_minority: 0.4492 - val_loss: 0.7561
Epoch 20/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7520 - loss: 0.7141 - val_f1_minority: 0.4463 - val_loss: 0.7544
Epoch 21/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7585 - loss: 0.7130 - val_f1_minority: 0.4426 - val_loss: 0.7661
Epoch 22/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7545 - loss: 0.7057 - val_f1_minority: 0.4367 - val_loss: 0.7783
Epoch 23/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7554 - loss: 0.7099 - val_f1_minority: 0.4440 - val_loss: 0.7754
Epoch 24/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7635 - loss: 0.6940 - val_f1_minority: 0.4473 - val_loss: 0.7620
Epoch 25/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7573 - loss: 0.6948 - val_f1_minority: 0.4533 - val_loss: 0.7241
Epoch 26/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7634 - loss: 0.6957 - val_f1_minority: 0.4411 - val_loss: 0.7467
Epoch 27/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7659 - loss: 0.6715 - val_f1_minority: 0.4508 - val_loss: 0.7379
Epoch 28/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7610 - loss: 0.6921 - val_f1_minority: 0.4461 - val_loss: 0.7604
Epoch 29/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7753 - loss: 0.6693 - val_f1_minority: 0.4529 - val_loss: 0.7251
Epoch 30/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7637 - loss: 0.6756 - val_f1_minority: 0.4506 - val_loss: 0.7562
Epoch 31/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7685 - loss: 0.6712 - val_f1_minority: 0.4508 - val_loss: 0.7489
Epoch 32/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7679 - loss: 0.6755 - val_f1_minority: 0.4472 - val_loss: 0.7366
Epoch 33/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7799 - loss: 0.6616 - val_f1_minority: 0.4478 - val_loss: 0.7465
Epoch 34/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 89ms/step - f1_minority: 0.7713 - loss: 0.6557 - val_f1_minority: 0.4409 - val_loss: 0.7512
Epoch 35/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7701 - loss: 0.6673 - val_f1_minority: 0.4498 - val_loss: 0.7086
Epoch 36/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 89ms/step - f1_minority: 0.7689 - loss: 0.6642 - val_f1_minority: 0.4460 - val_loss: 0.7305
Epoch 37/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7683 - loss: 0.6573 - val_f1_minority: 0.4492 - val_loss: 0.7272
Epoch 38/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 89ms/step - f1_minority: 0.7640 - loss: 0.6598 - val_f1_minority: 0.4526 - val_loss: 0.7316
Epoch 39/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7686 - loss: 0.6618 - val_f1_minority: 0.4497 - val_loss: 0.7461
Epoch 40/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7680 - loss: 0.6513 - val_f1_minority: 0.4505 - val_loss: 0.7243
Epoch 41/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7773 - loss: 0.6408 - val_f1_minority: 0.4420 - val_loss: 0.7539
Epoch 42/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 88ms/step - f1_minority: 0.7636 - loss: 0.6563 - val_f1_minority: 0.4438 - val_loss: 0.7698
Epoch 43/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7807 - loss: 0.6462 - val_f1_minority: 0.4409 - val_loss: 0.7685
Epoch 44/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7702 - loss: 0.6436 - val_f1_minority: 0.4463 - val_loss: 0.7264
Epoch 45/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7667 - loss: 0.6388 - val_f1_minority: 0.4508 - val_loss: 0.7214
Time taken in seconds  2564.5972447395325
In [134]:
plot(history,'loss')
No description has been provided for this image
In [136]:
plot(history,'f1_minority')
No description has been provided for this image
In [137]:
val_pred = model_15.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/50 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step
              precision    recall  f1-score   support

         0.0       0.95      0.55      0.70      1274
         1.0       0.33      0.88      0.48       326

    accuracy                           0.62      1600
   macro avg       0.64      0.71      0.59      1600
weighted avg       0.82      0.62      0.65      1600

No description has been provided for this image

not bad but i want to try more agressive class weights

more regulizer layers lower patience lower learning rate higher momentum and even more agressive class weights

In [143]:
import warnings
warnings.filterwarnings("ignore", category=UserWarning, module="keras")
In [146]:
model_16 = Sequential()

model_16.add(Input(shape=(X_train_smote.shape[1],)))
model_16.add(Dense(128, activation=LeakyReLU(negative_slope=0.1),  # Use negative_slope
                    kernel_initializer='glorot_uniform',
                    kernel_regularizer=regularizers.l2(0.001)))
model_16.add(BatchNormalization())
model_16.add(Dropout(0.5))

model_16.add(Dense(64, activation=LeakyReLU(negative_slope=0.1),  # Use negative_slope
                    kernel_initializer='glorot_uniform',
                    kernel_regularizer=regularizers.l2(0.001)))
model_16.add(BatchNormalization())
model_16.add(Dropout(0.5))

model_16.add(Dense(32, activation='relu',
                    kernel_initializer='glorot_uniform',
                    kernel_regularizer=regularizers.l2(0.001)))
model_16.add(BatchNormalization())
model_16.add(Dropout(0.5))

model_16.add(Dense(1, activation='sigmoid'))

#f1_minority
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_16.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[f1_minority])

early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

start = time.time()
history = model_16.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=16, epochs=100,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 86ms/step - f1_minority: 0.5806 - loss: 1.1193
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.5807 - loss: 1.1191 - val_f1_minority: 0.3761 - val_loss: 0.9355
Epoch 2/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.6936 - loss: 0.8861 - val_f1_minority: 0.4011 - val_loss: 0.8910
Epoch 3/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7009 - loss: 0.8579 - val_f1_minority: 0.4014 - val_loss: 0.8779
Epoch 4/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7145 - loss: 0.8463 - val_f1_minority: 0.4115 - val_loss: 0.8730
Epoch 5/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7319 - loss: 0.8179 - val_f1_minority: 0.4236 - val_loss: 0.8467
Epoch 6/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 89ms/step - f1_minority: 0.7291 - loss: 0.8179 - val_f1_minority: 0.4161 - val_loss: 0.8509
Epoch 7/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7318 - loss: 0.8117 - val_f1_minority: 0.4323 - val_loss: 0.8470
Epoch 8/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7337 - loss: 0.7934 - val_f1_minority: 0.4279 - val_loss: 0.8326
Epoch 9/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7480 - loss: 0.7689 - val_f1_minority: 0.4365 - val_loss: 0.8110
Epoch 10/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7408 - loss: 0.7812 - val_f1_minority: 0.4284 - val_loss: 0.8193
Epoch 11/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7454 - loss: 0.7751 - val_f1_minority: 0.4319 - val_loss: 0.8110
Epoch 12/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7529 - loss: 0.7551 - val_f1_minority: 0.4395 - val_loss: 0.8000
Epoch 13/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7430 - loss: 0.7604 - val_f1_minority: 0.4522 - val_loss: 0.7826
Epoch 14/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7571 - loss: 0.7442 - val_f1_minority: 0.4455 - val_loss: 0.7911
Epoch 15/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7468 - loss: 0.7522 - val_f1_minority: 0.4499 - val_loss: 0.7700
Epoch 16/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7583 - loss: 0.7192 - val_f1_minority: 0.4379 - val_loss: 0.7976
Epoch 17/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7646 - loss: 0.7195 - val_f1_minority: 0.4387 - val_loss: 0.7910
Epoch 18/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7576 - loss: 0.7252 - val_f1_minority: 0.4435 - val_loss: 0.7734
Epoch 19/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7662 - loss: 0.7018 - val_f1_minority: 0.4429 - val_loss: 0.7511
Epoch 20/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7627 - loss: 0.7083 - val_f1_minority: 0.4380 - val_loss: 0.7873
Epoch 21/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7564 - loss: 0.7084 - val_f1_minority: 0.4403 - val_loss: 0.7926
Epoch 22/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7601 - loss: 0.7032 - val_f1_minority: 0.4445 - val_loss: 0.7622
Epoch 23/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7571 - loss: 0.7087 - val_f1_minority: 0.4451 - val_loss: 0.7670
Epoch 24/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7637 - loss: 0.6925 - val_f1_minority: 0.4396 - val_loss: 0.7681
Epoch 25/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7710 - loss: 0.6928 - val_f1_minority: 0.4419 - val_loss: 0.7659
Epoch 26/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7711 - loss: 0.6761 - val_f1_minority: 0.4420 - val_loss: 0.7681
Epoch 27/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7610 - loss: 0.6906 - val_f1_minority: 0.4404 - val_loss: 0.7866
Epoch 28/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7592 - loss: 0.6856 - val_f1_minority: 0.4386 - val_loss: 0.7874
Epoch 29/100
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7651 - loss: 0.6864 - val_f1_minority: 0.4435 - val_loss: 0.7734
Time taken in seconds  1647.1443145275116
In [147]:
plot(history,'loss')
No description has been provided for this image
In [149]:
plot(history,'f1_minority')
No description has been provided for this image
In [175]:
val_pred = model_16.predict(X_val)
val_pred = (val_pred > 0.50)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/50 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step
              precision    recall  f1-score   support

         0.0       0.95      0.51      0.67      1274
         1.0       0.32      0.90      0.47       326

    accuracy                           0.59      1600
   macro avg       0.64      0.71      0.57      1600
weighted avg       0.83      0.59      0.63      1600

No description has been provided for this image
In [ ]:
df.shape
In [177]:
tf.keras.backend.clear_session()


model_17 = Sequential()

model_17.add(Dense(64, kernel_regularizer=regularizers.l2(0.001),
                    kernel_initializer='glorot_uniform', input_dim=X_train.shape[1]))  #not using smote or scaled
model_17.add(LeakyReLU(alpha=0.1))
model_17.add(BatchNormalization())
model_17.add(Dropout(0.4))

model_17.add(Dense(32, kernel_regularizer=regularizers.l2(0.001),
                    kernel_initializer='glorot_uniform'))
model_17.add(LeakyReLU(alpha=0.1))
model_17.add(BatchNormalization())
model_17.add(Dropout(0.4))

model_17.add(Dense(1, activation='sigmoid'))

class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: 1, 1: 2}

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.99)

model_17.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy']) #accuracy but with aggressive class weights and no smote

early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

start = time.time()
history = model_17.fit(X_train, y_train,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=8, epochs=25,  # Adjusted batch size
                    callbacks=[early_stopping])
Epoch 1/25
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
800/800 ━━━━━━━━━━━━━━━━━━━━ 0s 64ms/step - accuracy: 0.6619 - loss: 1.5933
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6619 - loss: 1.5936 - val_accuracy: 0.7875 - val_loss: 1.1769
Epoch 2/25
800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6587 - loss: 2.0130 - val_accuracy: 0.7025 - val_loss: 1.0165
Epoch 3/25
800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6745 - loss: 1.4768 - val_accuracy: 0.7856 - val_loss: 0.9387
Epoch 4/25
800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6611 - loss: 1.4657 - val_accuracy: 0.7600 - val_loss: 1.2144
Epoch 5/25
800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6827 - loss: 1.9323 - val_accuracy: 0.7956 - val_loss: 1.5615
Epoch 6/25
800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 68ms/step - accuracy: 0.6458 - loss: 3.0731 - val_accuracy: 0.7106 - val_loss: 1.7047
Epoch 7/25
800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 68ms/step - accuracy: 0.6573 - loss: 3.6581 - val_accuracy: 0.7744 - val_loss: 5.0377
Epoch 8/25
800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6587 - loss: 9.9355 - val_accuracy: 0.7013 - val_loss: 6.3116
In [178]:
plot(history,'loss')
No description has been provided for this image
In [179]:
plot(history,'accuracy')
No description has been provided for this image
In [180]:
val_pred = model_17.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
15/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
              precision    recall  f1-score   support

         0.0       0.80      0.98      0.88      1274
         1.0       0.24      0.02      0.04       326

    accuracy                           0.79      1600
   macro avg       0.52      0.50      0.46      1600
weighted avg       0.68      0.79      0.71      1600

No description has been provided for this image

smaller regularization larger learning rate smaller momentum and smaller dropout

In [156]:
tf.keras.backend.clear_session()

# Define macro F1-score metric
def macro_f1(y_true, y_pred):
    y_pred = K.round(y_pred)  # Round predictions to 0 or 1
    tp = K.sum(K.cast(y_true * y_pred, 'float'), axis=0)
    fp = K.sum(K.cast((1 - y_true) * y_pred, 'float'), axis=0)
    fn = K.sum(K.cast(y_true * (1 - y_pred), 'float'), axis=0)

    p = tp / (tp + fp + K.epsilon())
    r = tp / (tp + fn + K.epsilon())

    f1 = 2 * p * r / (p + r + K.epsilon())
    macro_f1 = K.mean(f1)

    return macro_f1
In [158]:
tf.keras.backend.clear_session()
# Define model_18 (renamed from model_20)
model_18 = Sequential()

# Input layer with L2 regularization and LeakyReLU activation
model_18.add(Dense(128, activation=LeakyReLU(alpha=0.05),
                    input_dim=X_train_smote.shape[1],
                    kernel_regularizer=regularizers.l2(0.001)))
model_18.add(BatchNormalization())
model_18.add(Dropout(0.4))

# Hidden layer with LeakyReLU activation and L2 regularization
model_18.add(Dense(64, activation=LeakyReLU(alpha=0.05),
                    kernel_regularizer=regularizers.l2(0.001)))
model_18.add(BatchNormalization())
model_18.add(Dropout(0.4))

# Hidden layer with ReLU activation
model_18.add(Dense(32, activation='relu'))

model_18.add(Dense(1, activation='sigmoid'))

# Calculate class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}  # Using computed weights

# Compile the model with macro F1-score metric
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)  # Adam optimizer
model_18.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[macro_f1])

early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

start = time.time()
history = model_18.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=32, epochs=100,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 103ms/step - loss: 0.8649 - macro_f1: 0.6796
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 106ms/step - loss: 0.8647 - macro_f1: 0.6796 - val_loss: 1.0137 - val_macro_f1: 0.4021
Epoch 2/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.7228 - macro_f1: 0.7120 - val_loss: 1.1489 - val_macro_f1: 0.4092
Epoch 3/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.6662 - macro_f1: 0.7414 - val_loss: 1.0782 - val_macro_f1: 0.4267
Epoch 4/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.6430 - macro_f1: 0.7505 - val_loss: 1.0401 - val_macro_f1: 0.4287
Epoch 5/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.6253 - macro_f1: 0.7543 - val_loss: 1.0120 - val_macro_f1: 0.4241
Epoch 6/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5975 - macro_f1: 0.7596 - val_loss: 1.0781 - val_macro_f1: 0.4315
Epoch 7/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5967 - macro_f1: 0.7664 - val_loss: 1.0448 - val_macro_f1: 0.4273
Epoch 8/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5736 - macro_f1: 0.7653 - val_loss: 1.1825 - val_macro_f1: 0.4193
Epoch 9/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5745 - macro_f1: 0.7631 - val_loss: 1.1427 - val_macro_f1: 0.4212
Epoch 10/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5576 - macro_f1: 0.7760 - val_loss: 1.1048 - val_macro_f1: 0.4261
Epoch 11/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 108ms/step - loss: 0.5701 - macro_f1: 0.7656 - val_loss: 1.0891 - val_macro_f1: 0.4318
Epoch 12/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 35s 109ms/step - loss: 0.5521 - macro_f1: 0.7775 - val_loss: 1.0942 - val_macro_f1: 0.4297
Epoch 13/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5439 - macro_f1: 0.7803 - val_loss: 1.1071 - val_macro_f1: 0.4302
Epoch 14/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5460 - macro_f1: 0.7743 - val_loss: 1.0465 - val_macro_f1: 0.4372
Epoch 15/100
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5339 - macro_f1: 0.7845 - val_loss: 1.1456 - val_macro_f1: 0.4226
Time taken in seconds  512.9856388568878
In [161]:
plot(history,'loss')
No description has been provided for this image
In [165]:
plot(history,'macro_f1')
No description has been provided for this image
In [166]:
val_pred = model_18.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
13/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
              precision    recall  f1-score   support

         0.0       0.97      0.39      0.56      1274
         1.0       0.29      0.95      0.44       326

    accuracy                           0.51      1600
   macro avg       0.63      0.67      0.50      1600
weighted avg       0.83      0.51      0.53      1600

No description has been provided for this image

we need to use our best moddel to map accuracy using smote and our best parameters and hyperparameters

In [32]:
tf.keras.backend.clear_session()

model_19 = Sequential()

model_19.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train_smote.shape[1],
                 kernel_regularizer=regularizers.l2(0.0001)))
model_19.add(Dropout(0.2))
model_19.add(Dense(64, activation=LeakyReLU(alpha=0.1)))
model_19.add(Dropout(0.2))
model_19.add(Dense(32, activation='relu'))
model_19.add(Dense(1, activation='sigmoid'))


class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)

model_19.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])

early_stopping = EarlyStopping(monitor='val_accuracy', patience=5, restore_best_weights=True)

start = time.time()
history = model_19.fit(X_train_smote_scaled, y_train_smote,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=16, epochs=50,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step - accuracy: 0.4996 - loss: 0.8184
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.4996 - loss: 0.8183 - val_accuracy: 0.2556 - val_loss: 1.0267
Epoch 2/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.5576 - loss: 0.6607 - val_accuracy: 0.3581 - val_loss: 0.9696
Epoch 3/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.6182 - loss: 0.6301 - val_accuracy: 0.3994 - val_loss: 0.9642
Epoch 4/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.6277 - loss: 0.6154 - val_accuracy: 0.4081 - val_loss: 1.0050
Epoch 5/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.6480 - loss: 0.5901 - val_accuracy: 0.4594 - val_loss: 0.9826
Epoch 6/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 31s 48ms/step - accuracy: 0.6771 - loss: 0.5677 - val_accuracy: 0.4712 - val_loss: 0.9927
Epoch 7/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7015 - loss: 0.5447 - val_accuracy: 0.4956 - val_loss: 0.9915
Epoch 8/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7130 - loss: 0.5358 - val_accuracy: 0.4913 - val_loss: 1.0247
Epoch 9/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7084 - loss: 0.5357 - val_accuracy: 0.4994 - val_loss: 1.0145
Epoch 10/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7149 - loss: 0.5233 - val_accuracy: 0.5063 - val_loss: 0.9989
Epoch 11/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7282 - loss: 0.5147 - val_accuracy: 0.4806 - val_loss: 1.0806
Epoch 12/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7188 - loss: 0.5170 - val_accuracy: 0.5006 - val_loss: 1.0140
Epoch 13/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7352 - loss: 0.5021 - val_accuracy: 0.5069 - val_loss: 1.0302
Epoch 14/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 31s 48ms/step - accuracy: 0.7315 - loss: 0.5003 - val_accuracy: 0.4888 - val_loss: 1.1102
Epoch 15/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7294 - loss: 0.5086 - val_accuracy: 0.5250 - val_loss: 0.9752
Epoch 16/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 31s 48ms/step - accuracy: 0.7360 - loss: 0.5005 - val_accuracy: 0.5050 - val_loss: 1.0546
Epoch 17/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7278 - loss: 0.5042 - val_accuracy: 0.5150 - val_loss: 1.0319
Epoch 18/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7393 - loss: 0.4947 - val_accuracy: 0.5150 - val_loss: 0.9952
Epoch 19/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7459 - loss: 0.4815 - val_accuracy: 0.5113 - val_loss: 1.0089
Epoch 20/50
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7356 - loss: 0.4971 - val_accuracy: 0.5088 - val_loss: 1.0783
Time taken in seconds  606.9027745723724
In [33]:
plot(history,'loss')
No description has been provided for this image
In [34]:
plot(history,'accuracy')
No description has been provided for this image
In [35]:
val_pred = model_19.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
28/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
              precision    recall  f1-score   support

         0.0       0.96      0.42      0.58      1274
         1.0       0.29      0.94      0.45       326

    accuracy                           0.53      1600
   macro avg       0.63      0.68      0.52      1600
weighted avg       0.83      0.53      0.56      1600

No description has been provided for this image
In [182]:
tf.keras.backend.clear_session()

model_20 = Sequential()
model_20.add(Dense(64, activation="tanh", input_dim=X_train.shape[1])) #tanh
model_20.add(Dense(32, activation="tanh"))#tanh
model_20.add(Dense(1, activation='sigmoid'))

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model_20.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])

early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

start = time.time()
history = model_20.fit(X_train, y_train,
                    validation_data=(X_val, y_val),
                    batch_size=32, epochs=50,
                    callbacks=[early_stopping])
end = time.time()
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
199/200 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step - accuracy: 0.7803 - loss: 0.4802
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.7805 - loss: 0.4798 - val_accuracy: 0.8138 - val_loss: 0.4177
Epoch 2/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.8251 - loss: 0.4018 - val_accuracy: 0.8281 - val_loss: 0.3841
Epoch 3/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.8383 - loss: 0.3783 - val_accuracy: 0.8413 - val_loss: 0.3632
Epoch 4/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.8515 - loss: 0.3559 - val_accuracy: 0.8469 - val_loss: 0.3548
Epoch 5/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8621 - loss: 0.3443 - val_accuracy: 0.8506 - val_loss: 0.3505
Epoch 6/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8638 - loss: 0.3485 - val_accuracy: 0.8519 - val_loss: 0.3480
Epoch 7/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8588 - loss: 0.3418 - val_accuracy: 0.8519 - val_loss: 0.3487
Epoch 8/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.8596 - loss: 0.3353 - val_accuracy: 0.8537 - val_loss: 0.3469
Epoch 9/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8647 - loss: 0.3264 - val_accuracy: 0.8519 - val_loss: 0.3463
Epoch 10/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 51ms/step - accuracy: 0.8608 - loss: 0.3353 - val_accuracy: 0.8512 - val_loss: 0.3510
Epoch 11/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8606 - loss: 0.3336 - val_accuracy: 0.8500 - val_loss: 0.3541
Epoch 12/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 49ms/step - accuracy: 0.8635 - loss: 0.3378 - val_accuracy: 0.8481 - val_loss: 0.3467
Epoch 13/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 51ms/step - accuracy: 0.8666 - loss: 0.3351 - val_accuracy: 0.8512 - val_loss: 0.3491
Epoch 14/50
200/200 ━━━━━━━━━━━━━━━━━━━━ 11s 53ms/step - accuracy: 0.8698 - loss: 0.3239 - val_accuracy: 0.8587 - val_loss: 0.3485
In [186]:
plot(history,'loss')
No description has been provided for this image
In [187]:
plot(history,'accuracy')
No description has been provided for this image
In [183]:
val_pred = model_20.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
25/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
              precision    recall  f1-score   support

         0.0       0.87      0.96      0.91      1274
         1.0       0.73      0.44      0.55       326

    accuracy                           0.85      1600
   macro avg       0.80      0.70      0.73      1600
weighted avg       0.84      0.85      0.84      1600

No description has been provided for this image

not using the x_train_smote/scaled variables were the problem but we need to keep going because the recall is bad but now the accuracy is good

In [189]:
tf.keras.backend.clear_session()

model_21 = Sequential()

model_21.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train.shape[1],
                 kernel_regularizer=regularizers.l2(0.0001)))
model_21.add(Dropout(0.2))
model_21.add(Dense(64, activation=LeakyReLU(alpha=0.1)))
model_21.add(Dropout(0.2))
model_21.add(Dense(32, activation='relu'))
model_21.add(Dense(1, activation='sigmoid'))


class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}

optimizer = tf.keras.optimizers.Adam()

model_21.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])

early_stopping = EarlyStopping(monitor='val_accuracy', patience=7, restore_best_weights=True)

start = time.time()
history = model_21.fit(X_train, y_train,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=16, epochs=50,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 0s 72ms/step - accuracy: 0.7003 - loss: 0.6147
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 77ms/step - accuracy: 0.7003 - loss: 0.6146 - val_accuracy: 0.7831 - val_loss: 0.4704
Epoch 2/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 76ms/step - accuracy: 0.7577 - loss: 0.4956 - val_accuracy: 0.7594 - val_loss: 0.4859
Epoch 3/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 76ms/step - accuracy: 0.7714 - loss: 0.4891 - val_accuracy: 0.7656 - val_loss: 0.4809
Epoch 4/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 76ms/step - accuracy: 0.7852 - loss: 0.4585 - val_accuracy: 0.7844 - val_loss: 0.4795
Epoch 5/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8118 - loss: 0.4496 - val_accuracy: 0.7819 - val_loss: 0.4761
Epoch 6/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7971 - loss: 0.4517 - val_accuracy: 0.7688 - val_loss: 0.4876
Epoch 7/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7948 - loss: 0.4543 - val_accuracy: 0.7925 - val_loss: 0.4620
Epoch 8/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8072 - loss: 0.4560 - val_accuracy: 0.8050 - val_loss: 0.4427
Epoch 9/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7937 - loss: 0.4708 - val_accuracy: 0.7613 - val_loss: 0.5064
Epoch 10/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7942 - loss: 0.4534 - val_accuracy: 0.7950 - val_loss: 0.4471
Epoch 11/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7937 - loss: 0.4476 - val_accuracy: 0.8100 - val_loss: 0.4328
Epoch 12/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8038 - loss: 0.4510 - val_accuracy: 0.7944 - val_loss: 0.4421
Epoch 13/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8031 - loss: 0.4452 - val_accuracy: 0.8081 - val_loss: 0.4455
Epoch 14/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8047 - loss: 0.4380 - val_accuracy: 0.7875 - val_loss: 0.4603
Epoch 15/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8021 - loss: 0.4450 - val_accuracy: 0.7875 - val_loss: 0.4726
Epoch 16/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8038 - loss: 0.4406 - val_accuracy: 0.7725 - val_loss: 0.4859
Epoch 17/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8014 - loss: 0.4373 - val_accuracy: 0.7875 - val_loss: 0.4707
Epoch 18/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7966 - loss: 0.4353 - val_accuracy: 0.8169 - val_loss: 0.4269
Epoch 19/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8067 - loss: 0.4344 - val_accuracy: 0.7844 - val_loss: 0.4732
Epoch 20/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8101 - loss: 0.4270 - val_accuracy: 0.7638 - val_loss: 0.4789
Epoch 21/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8172 - loss: 0.4187 - val_accuracy: 0.7900 - val_loss: 0.4585
Epoch 22/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8053 - loss: 0.4317 - val_accuracy: 0.7944 - val_loss: 0.4613
Epoch 23/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8021 - loss: 0.4286 - val_accuracy: 0.8050 - val_loss: 0.4290
Epoch 24/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8042 - loss: 0.4289 - val_accuracy: 0.7875 - val_loss: 0.4645
Epoch 25/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8014 - loss: 0.4323 - val_accuracy: 0.7513 - val_loss: 0.5132
Time taken in seconds  747.0243382453918
In [190]:
plot(history,'loss')
No description has been provided for this image
In [191]:
plot(history,'accuracy')
No description has been provided for this image

good but we need a lower learning rate more epoch and potentailly some more tuning that we already refined on the other x's and y's

In [192]:
val_pred = model_21.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
19/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
              precision    recall  f1-score   support

         0.0       0.92      0.84      0.88      1274
         1.0       0.54      0.71      0.61       326

    accuracy                           0.82      1600
   macro avg       0.73      0.78      0.75      1600
weighted avg       0.84      0.82      0.83      1600

No description has been provided for this image

these results are good just too much oscilation

In [194]:
tf.keras.backend.clear_session()
# Define model_18 (renamed from model_20)
model_22 = Sequential()

# Input layer with L2 regularization and LeakyReLU activation
model_22.add(Dense(128, activation=LeakyReLU(alpha=0.05),
                    input_dim=X_train.shape[1],
                    kernel_regularizer=regularizers.l2(0.001)))
model_22.add(BatchNormalization())
model_22.add(Dropout(0.4))

# Hidden layer with LeakyReLU activation and L2 regularization
model_22.add(Dense(64, activation=LeakyReLU(alpha=0.05),
                    kernel_regularizer=regularizers.l2(0.001)))
model_22.add(BatchNormalization())
model_22.add(Dropout(0.4))

# Hidden layer with ReLU activation
model_22.add(Dense(32, activation='relu'))

model_22.add(Dense(1, activation='sigmoid'))

# Calculate class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}  # Using computed weights

# Compile the model with macro F1-score metric
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)  # Adam optimizer
model_22.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[macro_f1])

early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

start = time.time()
history = model_22.fit(X_train, y_train,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=32, epochs=100,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 105ms/step - loss: 0.8218 - macro_f1: 0.3913
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.8214 - macro_f1: 0.3915 - val_loss: 0.6059 - val_macro_f1: 0.5133
Epoch 2/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.6514 - macro_f1: 0.4964 - val_loss: 0.5332 - val_macro_f1: 0.5527
Epoch 3/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.6473 - macro_f1: 0.5358 - val_loss: 0.5341 - val_macro_f1: 0.5680
Epoch 4/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 112ms/step - loss: 0.6084 - macro_f1: 0.5207 - val_loss: 0.5449 - val_macro_f1: 0.5680
Epoch 5/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.6105 - macro_f1: 0.5427 - val_loss: 0.5374 - val_macro_f1: 0.5786
Epoch 6/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5971 - macro_f1: 0.5425 - val_loss: 0.5336 - val_macro_f1: 0.5543
Epoch 7/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5860 - macro_f1: 0.5592 - val_loss: 0.5114 - val_macro_f1: 0.5732
Epoch 8/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5721 - macro_f1: 0.5617 - val_loss: 0.5107 - val_macro_f1: 0.5749
Epoch 9/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5725 - macro_f1: 0.5583 - val_loss: 0.5290 - val_macro_f1: 0.5662
Epoch 10/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5242 - macro_f1: 0.5773 - val_loss: 0.5123 - val_macro_f1: 0.5714
Epoch 11/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5478 - macro_f1: 0.5695 - val_loss: 0.5117 - val_macro_f1: 0.5737
Epoch 12/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5396 - macro_f1: 0.5884 - val_loss: 0.5052 - val_macro_f1: 0.5868
Epoch 13/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 112ms/step - loss: 0.5155 - macro_f1: 0.5862 - val_loss: 0.4876 - val_macro_f1: 0.5849
Epoch 14/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 109ms/step - loss: 0.5100 - macro_f1: 0.5678 - val_loss: 0.4919 - val_macro_f1: 0.5911
Epoch 15/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5194 - macro_f1: 0.5736 - val_loss: 0.4913 - val_macro_f1: 0.5886
Epoch 16/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5303 - macro_f1: 0.5809 - val_loss: 0.4704 - val_macro_f1: 0.5874
Epoch 17/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5052 - macro_f1: 0.6090 - val_loss: 0.4920 - val_macro_f1: 0.5931
Epoch 18/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 23s 112ms/step - loss: 0.5138 - macro_f1: 0.5832 - val_loss: 0.4952 - val_macro_f1: 0.5728
Epoch 19/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 112ms/step - loss: 0.5190 - macro_f1: 0.5798 - val_loss: 0.4867 - val_macro_f1: 0.5726
Epoch 20/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.4949 - macro_f1: 0.5737 - val_loss: 0.4892 - val_macro_f1: 0.5585
Epoch 21/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5203 - macro_f1: 0.5758 - val_loss: 0.4801 - val_macro_f1: 0.5798
Epoch 22/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 109ms/step - loss: 0.5031 - macro_f1: 0.5730 - val_loss: 0.4957 - val_macro_f1: 0.5860
Epoch 23/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 109ms/step - loss: 0.5026 - macro_f1: 0.5720 - val_loss: 0.4879 - val_macro_f1: 0.5823
Epoch 24/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.4985 - macro_f1: 0.5729 - val_loss: 0.4967 - val_macro_f1: 0.5763
Epoch 25/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 109ms/step - loss: 0.4826 - macro_f1: 0.5834 - val_loss: 0.4803 - val_macro_f1: 0.5772
Epoch 26/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.4904 - macro_f1: 0.5628 - val_loss: 0.4847 - val_macro_f1: 0.5839
Time taken in seconds  575.4505610466003
In [195]:
plot(history,'loss')
No description has been provided for this image
In [196]:
plot(history,'macro_f1')
No description has been provided for this image
In [197]:
val_pred = model_22.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
13/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step
              precision    recall  f1-score   support

         0.0       0.92      0.82      0.87      1274
         1.0       0.52      0.74      0.61       326

    accuracy                           0.81      1600
   macro avg       0.72      0.78      0.74      1600
weighted avg       0.84      0.81      0.82      1600

No description has been provided for this image

slightly worse performance but fewer oscilations

In [206]:
tf.keras.backend.clear_session()
# Define model_18 (renamed from model_20)
model_23 = Sequential()

# Input layer with L2 regularization and LeakyReLU activation
model_23.add(Dense(64, activation=LeakyReLU(alpha=0.05),
                    input_dim=X_train.shape[1],
                    kernel_regularizer=regularizers.l2(0.001)))
model_23.add(BatchNormalization())
model_23.add(Dropout(0.2))

# Hidden layer with LeakyReLU activation and L2 regularization
model_23.add(Dense(32, activation=LeakyReLU(alpha=0.05),
                    kernel_regularizer=regularizers.l2(0.001)))
model_23.add(BatchNormalization())
model_23.add(Dropout(0.2))

model_23.add(Dense(16, activation=LeakyReLU(alpha=0.05),
                    kernel_regularizer=regularizers.l2(0.001)))
model_23.add(BatchNormalization())
model_23.add(Dropout(0.2))
model_23.add(Dense(1, activation='sigmoid'))

# Calculate class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}  # Using computed weights

# Compile the model with macro F1-score metric
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)  # Adam optimizer
model_23.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)  # Adam optimizer
model_23.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])

early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

start = time.time()
history = model_23.fit(X_train, y_train,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=32, epochs=100,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 127ms/step - accuracy: 0.5666 - loss: 0.9134
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.5668 - loss: 0.9128 - val_accuracy: 0.7819 - val_loss: 0.6017
Epoch 2/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 136ms/step - accuracy: 0.6775 - loss: 0.7094 - val_accuracy: 0.7875 - val_loss: 0.5680
Epoch 3/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 136ms/step - accuracy: 0.7156 - loss: 0.6501 - val_accuracy: 0.7969 - val_loss: 0.5429
Epoch 4/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7293 - loss: 0.6135 - val_accuracy: 0.7788 - val_loss: 0.5509
Epoch 5/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7317 - loss: 0.6280 - val_accuracy: 0.7931 - val_loss: 0.5280
Epoch 6/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 136ms/step - accuracy: 0.7446 - loss: 0.6027 - val_accuracy: 0.7994 - val_loss: 0.5195
Epoch 7/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7442 - loss: 0.5820 - val_accuracy: 0.8019 - val_loss: 0.5174
Epoch 8/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7538 - loss: 0.5780 - val_accuracy: 0.8012 - val_loss: 0.5144
Epoch 9/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7599 - loss: 0.5753 - val_accuracy: 0.7788 - val_loss: 0.5293
Epoch 10/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7622 - loss: 0.5727 - val_accuracy: 0.7981 - val_loss: 0.5164
Epoch 11/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7718 - loss: 0.5711 - val_accuracy: 0.7862 - val_loss: 0.5230
Epoch 12/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 130ms/step - accuracy: 0.7707 - loss: 0.5395 - val_accuracy: 0.7831 - val_loss: 0.5213
Epoch 13/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7730 - loss: 0.5358 - val_accuracy: 0.8025 - val_loss: 0.4969
Epoch 14/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7709 - loss: 0.5439 - val_accuracy: 0.7844 - val_loss: 0.5134
Epoch 15/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7662 - loss: 0.5376 - val_accuracy: 0.7994 - val_loss: 0.5040
Epoch 16/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7731 - loss: 0.5368 - val_accuracy: 0.7956 - val_loss: 0.4972
Epoch 17/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7688 - loss: 0.5256 - val_accuracy: 0.7906 - val_loss: 0.4961
Epoch 18/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7851 - loss: 0.5212 - val_accuracy: 0.7950 - val_loss: 0.5039
Epoch 19/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7926 - loss: 0.5000 - val_accuracy: 0.7944 - val_loss: 0.4968
Epoch 20/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7681 - loss: 0.5320 - val_accuracy: 0.8006 - val_loss: 0.4747
Epoch 21/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7806 - loss: 0.5133 - val_accuracy: 0.7850 - val_loss: 0.4904
Epoch 22/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7723 - loss: 0.5221 - val_accuracy: 0.7881 - val_loss: 0.5008
Epoch 23/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7717 - loss: 0.5285 - val_accuracy: 0.7825 - val_loss: 0.4961
Epoch 24/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7712 - loss: 0.5201 - val_accuracy: 0.7825 - val_loss: 0.4991
Epoch 25/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7853 - loss: 0.5223 - val_accuracy: 0.8069 - val_loss: 0.4723
Epoch 26/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7894 - loss: 0.4974 - val_accuracy: 0.8062 - val_loss: 0.4768
Epoch 27/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7753 - loss: 0.5096 - val_accuracy: 0.8000 - val_loss: 0.4773
Epoch 28/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7800 - loss: 0.5011 - val_accuracy: 0.7719 - val_loss: 0.4967
Epoch 29/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7707 - loss: 0.5067 - val_accuracy: 0.8094 - val_loss: 0.4715
Epoch 30/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 137ms/step - accuracy: 0.7808 - loss: 0.4989 - val_accuracy: 0.8037 - val_loss: 0.4773
Epoch 31/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7874 - loss: 0.4984 - val_accuracy: 0.7881 - val_loss: 0.4840
Epoch 32/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7877 - loss: 0.4763 - val_accuracy: 0.7962 - val_loss: 0.4717
Epoch 33/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7861 - loss: 0.4852 - val_accuracy: 0.8031 - val_loss: 0.4720
Epoch 34/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7856 - loss: 0.4966 - val_accuracy: 0.8006 - val_loss: 0.4632
Epoch 35/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7837 - loss: 0.4958 - val_accuracy: 0.7944 - val_loss: 0.4737
Epoch 36/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7821 - loss: 0.5083 - val_accuracy: 0.7987 - val_loss: 0.4710
Epoch 37/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7853 - loss: 0.5016 - val_accuracy: 0.7900 - val_loss: 0.4816
Epoch 38/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7845 - loss: 0.4903 - val_accuracy: 0.7962 - val_loss: 0.4696
Epoch 39/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7861 - loss: 0.4850 - val_accuracy: 0.7906 - val_loss: 0.4825
Epoch 40/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 132ms/step - accuracy: 0.7826 - loss: 0.4825 - val_accuracy: 0.8050 - val_loss: 0.4568
Epoch 41/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 132ms/step - accuracy: 0.7794 - loss: 0.4824 - val_accuracy: 0.7825 - val_loss: 0.4873
Epoch 42/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7804 - loss: 0.4924 - val_accuracy: 0.8050 - val_loss: 0.4624
Epoch 43/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7906 - loss: 0.5049 - val_accuracy: 0.7956 - val_loss: 0.4718
Epoch 44/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7919 - loss: 0.4742 - val_accuracy: 0.8012 - val_loss: 0.4710
Epoch 45/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7845 - loss: 0.4845 - val_accuracy: 0.7763 - val_loss: 0.4790
Epoch 46/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7832 - loss: 0.4694 - val_accuracy: 0.8025 - val_loss: 0.4571
Epoch 47/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7937 - loss: 0.4948 - val_accuracy: 0.7819 - val_loss: 0.4800
Epoch 48/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7875 - loss: 0.4751 - val_accuracy: 0.7994 - val_loss: 0.4709
Epoch 49/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7700 - loss: 0.4820 - val_accuracy: 0.7950 - val_loss: 0.4686
Epoch 50/100
200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7881 - loss: 0.4876 - val_accuracy: 0.7919 - val_loss: 0.4678
Time taken in seconds  1332.7293946743011
In [207]:
plot(history,'loss')
No description has been provided for this image
In [208]:
plot(history,'accuracy')
No description has been provided for this image
In [209]:
val_pred = model_23.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/50 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step
              precision    recall  f1-score   support

         0.0       0.92      0.83      0.87      1274
         1.0       0.52      0.72      0.60       326

    accuracy                           0.81      1600
   macro avg       0.72      0.77      0.74      1600
weighted avg       0.84      0.81      0.82      1600

No description has been provided for this image
In [210]:
tf.keras.backend.clear_session()

model_24 = Sequential()

model_24.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train.shape[1],
                 kernel_regularizer=regularizers.l2(0.0001)))
model_24.add(Dropout(0.2))
model_24.add(Dense(64, activation=LeakyReLU(alpha=0.1)))
model_24.add(Dropout(0.2))
model_24.add(Dense(32, activation='relu'))
model_24.add(Dense(1, activation='sigmoid'))


class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}

optimizer = tf.keras.optimizers.Adam()

model_24.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])

early_stopping = EarlyStopping(monitor='val_accuracy', patience=7, restore_best_weights=True)

start = time.time()
history = model_24.fit(X_train, y_train,
                    validation_data=(X_val, y_val),
                    class_weight=class_weights_dict,
                    batch_size=16, epochs=50,
                    callbacks=[early_stopping])
end = time.time()

print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 0s 71ms/step - accuracy: 0.6154 - loss: 0.6186
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 76ms/step - accuracy: 0.6156 - loss: 0.6185 - val_accuracy: 0.7375 - val_loss: 0.5140
Epoch 2/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 78ms/step - accuracy: 0.7518 - loss: 0.5142 - val_accuracy: 0.7900 - val_loss: 0.4582
Epoch 3/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 77ms/step - accuracy: 0.7776 - loss: 0.4878 - val_accuracy: 0.7481 - val_loss: 0.5360
Epoch 4/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 78ms/step - accuracy: 0.7786 - loss: 0.4882 - val_accuracy: 0.8006 - val_loss: 0.4557
Epoch 5/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 76ms/step - accuracy: 0.7995 - loss: 0.4749 - val_accuracy: 0.7944 - val_loss: 0.4598
Epoch 6/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 76ms/step - accuracy: 0.7996 - loss: 0.4568 - val_accuracy: 0.7975 - val_loss: 0.4542
Epoch 7/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.7945 - loss: 0.4586 - val_accuracy: 0.8256 - val_loss: 0.4129
Epoch 8/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8076 - loss: 0.4520 - val_accuracy: 0.7944 - val_loss: 0.4638
Epoch 9/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 76ms/step - accuracy: 0.8098 - loss: 0.4400 - val_accuracy: 0.8094 - val_loss: 0.4458
Epoch 10/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.7960 - loss: 0.4572 - val_accuracy: 0.8163 - val_loss: 0.4253
Epoch 11/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 77ms/step - accuracy: 0.7972 - loss: 0.4443 - val_accuracy: 0.7925 - val_loss: 0.4552
Epoch 12/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 77ms/step - accuracy: 0.7976 - loss: 0.4464 - val_accuracy: 0.7900 - val_loss: 0.4665
Epoch 13/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 78ms/step - accuracy: 0.8120 - loss: 0.4311 - val_accuracy: 0.8144 - val_loss: 0.4249
Epoch 14/50
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8093 - loss: 0.4458 - val_accuracy: 0.7962 - val_loss: 0.4533
Time taken in seconds  427.6271617412567
In [211]:
plot(history,'loss')
No description has been provided for this image
In [212]:
plot(history,'accuracy')
No description has been provided for this image
In [213]:
val_pred = model_24.predict(X_val)
val_pred = (val_pred > 0.5)  # Convert probabilities to binary predictions (0 or 1)

# Print the classification report
print(classification_report(y_val, val_pred))

# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
28/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
              precision    recall  f1-score   support

         0.0       0.91      0.87      0.89      1274
         1.0       0.56      0.64      0.60       326

    accuracy                           0.83      1600
   macro avg       0.73      0.76      0.74      1600
weighted avg       0.84      0.83      0.83      1600

No description has been provided for this image

despoite our valient efforts we have failed to satisfy risk management and the marklleting department simultaneously with a single model.

Model Performance Comparison and Final Model Selection¶

the scaled smote training varioables messed up a lot of models buy by applying classs weights and using accuracy we were able to build decent models. of these model 23 had the best balance of performance and smoothness

In [214]:
# Generate predictions on the test set
test_pred = model_23.predict(X_test)
test_pred = (test_pred > 0.5)  # Convert probabilities to binary predictions

# Print the classification report
print(classification_report(y_test, test_pred))

# Create the confusion matrix
cm = confusion_matrix(y_test, test_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/63 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
63/63 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step
              precision    recall  f1-score   support

         0.0       0.92      0.82      0.86      1593
         1.0       0.50      0.71      0.58       407

    accuracy                           0.79      2000
   macro avg       0.71      0.76      0.72      2000
weighted avg       0.83      0.79      0.81      2000

No description has been provided for this image

very comprable performance. it may not be what we were hoping for but its not bad. it also flags a significant number of non-churners as potential churners

In [215]:
# Generate predictions on the test set
test_pred = model_23.predict(X_test)
test_pred = (test_pred > 0.3)  # Convert probabilities to binary predictions

# Print the classification report
print(classification_report(y_test, test_pred))

# Create the confusion matrix
cm = confusion_matrix(y_test, test_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/63 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
63/63 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step
              precision    recall  f1-score   support

         0.0       0.94      0.60      0.73      1593
         1.0       0.35      0.86      0.50       407

    accuracy                           0.65      2000
   macro avg       0.65      0.73      0.61      2000
weighted avg       0.82      0.65      0.68      2000

No description has been provided for this image

a larger portion of customers predicted as churners are actually not going to churn

In [216]:
# Generate predictions on the test set
test_pred = model_23.predict(X_test)
test_pred = (test_pred > 0.4)  # Convert probabilities to binary predictions

# Print the classification report
print(classification_report(y_test, test_pred))

# Create the confusion matrix
cm = confusion_matrix(y_test, test_pred)

# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
            xticklabels=['Not Churned', 'Churned'],
            yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/63 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(
63/63 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step
              precision    recall  f1-score   support

         0.0       0.93      0.72      0.81      1593
         1.0       0.42      0.79      0.55       407

    accuracy                           0.74      2000
   macro avg       0.68      0.76      0.68      2000
weighted avg       0.83      0.74      0.76      2000

No description has been provided for this image

we should stick with the .5 threshold... otherwise a lot of our errors move over too the very dangerous predicted churn actually not churn category

Actionable Insights and Business Recommendations¶

we need do develop retenntion starategies:

Offer incentives such as providing discounts or loyalty programs of different kinds.

ideally these offers should be personalized or segmented/clustered

Improve Customer Experience: Address potential pain points identified through the data analysis.

Proactive Communication: Reach out to these customers before they churn to understand their concerns and offer solutions.

we also need to do more Feature Engineering to improve model accuracy and reduce false positives.

collect more data

keep an eye on model performance

Power Ahead